2024-03-25T16:14:56.595 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-25T16:14:56.595 DEBUG:teuthology.run:Teuthology command: teuthology --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} --verbose --archive /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262 --owner scheduled_amathuri@teuthology --name amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi -- /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/orig.config.yaml 2024-03-25T16:14:56.640 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T16:14:56.730 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} email: null first_in_suite: false job_id: '7621262' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: mon client directed command retry: 5 mon election default strategy: 3 ms bind msgr2: false ms inject socket failures: 5000 ms type: async osd heartbeat grace: 80 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osd crush smoke test: false mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bdev async discard: true bdev enable discard: true bluestore allocator: stupid bluestore block size: 96636764160 bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 1 debug bluestore: 1 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd fast shutdown: false osd max markdown count: 1000 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - but it is still running - objects unfound and apparently lost - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request - overall HEALTH_ - \(PG_ - \(MON_DOWN\) - \(SLOW_OPS\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 valgrind: mds: - --tool=memcheck mon: - --tool=memcheck - --leak-check=full - --show-reachable=yes osd: - --tool=memcheck ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: debuginfo: true flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 6785 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi007.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNO605m0K+s3ytSEjhPuzWOnVvx5mM9qJa4VSjBMg+mUch9L3YOnpZK3XO1/G98YOVP8dPUsCh76nbIdEPo2i1s= smithi183.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCCm4RjM8duBU3f188Ut1weiOsvtzl+k29zkv296jFRM9XUdjgElzX0FBFJ83grLAIK4EwEXwcZIeVDmVJFqeMM= tasks: - install: null - ceph: null - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-25_13:28:33 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-25T16:14:56.730 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-25T16:14:56.731 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-25T16:14:56.731 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-25T16:14:56.732 INFO:teuthology.task.internal:Checking packages... 2024-03-25T16:14:56.756 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-25T16:14:56.756 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-25T16:14:56.756 INFO:teuthology.packaging:ref: None 2024-03-25T16:14:56.756 INFO:teuthology.packaging:tag: None 2024-03-25T16:14:56.756 INFO:teuthology.packaging:branch: main 2024-03-25T16:14:56.756 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:14:56.757 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-25T16:14:57.081 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2381.g44a9b520 2024-03-25T16:14:57.083 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-25T16:14:57.089 INFO:teuthology.task.internal:no buildpackages task found 2024-03-25T16:14:57.089 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-25T16:14:57.120 INFO:teuthology.task.internal:Saving configuration 2024-03-25T16:14:57.137 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-25T16:14:57.154 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T16:14:57.180 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:59.402534', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNO605m0K+s3ytSEjhPuzWOnVvx5mM9qJa4VSjBMg+mUch9L3YOnpZK3XO1/G98YOVP8dPUsCh76nbIdEPo2i1s='} 2024-03-25T16:14:57.204 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:59.404415', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCCm4RjM8duBU3f188Ut1weiOsvtzl+k29zkv296jFRM9XUdjgElzX0FBFJ83grLAIK4EwEXwcZIeVDmVJFqeMM='} 2024-03-25T16:14:57.204 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-25T16:14:57.210 INFO:teuthology.task.internal:roles: ubuntu@smithi007.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-25T16:14:57.210 INFO:teuthology.task.internal:roles: ubuntu@smithi183.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-25T16:14:57.210 INFO:teuthology.run_tasks:Running task console_log... 2024-03-25T16:14:57.309 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7ff1c81ff0d0>, signals=[15]) 2024-03-25T16:14:57.309 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-25T16:14:57.317 INFO:teuthology.task.internal:Opening connections... 2024-03-25T16:14:57.317 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T16:14:57.319 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:57.432 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T16:14:57.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:14:57.522 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-25T16:14:57.529 DEBUG:teuthology.orchestra.run.smithi007:> uname -m 2024-03-25T16:14:57.547 INFO:teuthology.orchestra.run.smithi007.stdout:x86_64 2024-03-25T16:14:57.547 DEBUG:teuthology.orchestra.run.smithi007:> cat /etc/os-release 2024-03-25T16:14:57.605 INFO:teuthology.orchestra.run.smithi007.stdout:NAME="CentOS Stream" 2024-03-25T16:14:57.605 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION="9" 2024-03-25T16:14:57.605 INFO:teuthology.orchestra.run.smithi007.stdout:ID="centos" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:VERSION_ID="9" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:ANSI_COLOR="0;31" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:14:57.606 INFO:teuthology.orchestra.run.smithi007.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:14:57.607 INFO:teuthology.lock.ops:Updating smithi007.front.sepia.ceph.com on lock server 2024-03-25T16:14:57.636 DEBUG:teuthology.orchestra.run.smithi183:> uname -m 2024-03-25T16:14:57.653 INFO:teuthology.orchestra.run.smithi183.stdout:x86_64 2024-03-25T16:14:57.653 DEBUG:teuthology.orchestra.run.smithi183:> cat /etc/os-release 2024-03-25T16:14:57.710 INFO:teuthology.orchestra.run.smithi183.stdout:NAME="CentOS Stream" 2024-03-25T16:14:57.710 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION="9" 2024-03-25T16:14:57.710 INFO:teuthology.orchestra.run.smithi183.stdout:ID="centos" 2024-03-25T16:14:57.710 INFO:teuthology.orchestra.run.smithi183.stdout:ID_LIKE="rhel fedora" 2024-03-25T16:14:57.710 INFO:teuthology.orchestra.run.smithi183.stdout:VERSION_ID="9" 2024-03-25T16:14:57.710 INFO:teuthology.orchestra.run.smithi183.stdout:PLATFORM_ID="platform:el9" 2024-03-25T16:14:57.710 INFO:teuthology.orchestra.run.smithi183.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-25T16:14:57.711 INFO:teuthology.orchestra.run.smithi183.stdout:ANSI_COLOR="0;31" 2024-03-25T16:14:57.711 INFO:teuthology.orchestra.run.smithi183.stdout:LOGO="fedora-logo-icon" 2024-03-25T16:14:57.711 INFO:teuthology.orchestra.run.smithi183.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-25T16:14:57.711 INFO:teuthology.orchestra.run.smithi183.stdout:HOME_URL="https://centos.org/" 2024-03-25T16:14:57.711 INFO:teuthology.orchestra.run.smithi183.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-25T16:14:57.711 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-25T16:14:57.711 INFO:teuthology.orchestra.run.smithi183.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-25T16:14:57.711 INFO:teuthology.lock.ops:Updating smithi183.front.sepia.ceph.com on lock server 2024-03-25T16:14:57.735 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-25T16:14:57.744 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-25T16:14:57.762 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-25T16:14:57.763 DEBUG:teuthology.orchestra.run.smithi007:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:14:57.765 DEBUG:teuthology.orchestra.run.smithi183:> test '!' -e /home/ubuntu/cephtest 2024-03-25T16:14:57.781 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-25T16:14:57.788 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-25T16:14:57.788 DEBUG:teuthology.orchestra.run.smithi007:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:14:57.823 DEBUG:teuthology.orchestra.run.smithi183:> test -z $(ls -A /var/lib/ceph) 2024-03-25T16:14:57.856 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-25T16:14:57.914 INFO:teuthology.run_tasks:Running task kernel... 2024-03-25T16:14:57.928 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-25T16:14:57.928 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-25T16:14:57.929 DEBUG:teuthology.orchestra.run.smithi007:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:14:57.929 DEBUG:teuthology.orchestra.run.smithi183:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-25T16:14:57.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:57.945 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-25T16:14:57.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:14:57.946 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-25T16:14:58.003 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:14:58.003 INFO:teuthology.task.kernel:Running kernel on smithi183: 5.14.0-361.el9.x86_64 2024-03-25T16:14:58.003 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum install -y kernel 2024-03-25T16:14:58.004 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-361.el9.x86_64 2024-03-25T16:14:58.004 INFO:teuthology.task.kernel:Running kernel on smithi007: 5.14.0-361.el9.x86_64 2024-03-25T16:14:58.005 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum install -y kernel 2024-03-25T16:15:00.088 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-25T16:15:00.180 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 11 MB/s | 6.7 MB 00:00 2024-03-25T16:15:01.630 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-25T16:15:02.197 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 2.2 MB/s | 2.0 MB 00:00 2024-03-25T16:15:03.696 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-25T16:15:04.079 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.0 MB 00:01 2024-03-25T16:15:05.988 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-25T16:15:07.022 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 8.4 MB/s | 19 MB 00:02 2024-03-25T16:15:10.156 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 8.4 MB/s | 6.1 MB 00:00 2024-03-25T16:15:11.530 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 6.4 MB/s | 6.1 MB 00:00 2024-03-25T16:15:12.431 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 14 kB/s | 16 kB 00:01 2024-03-25T16:15:12.973 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 16 kB 00:00 2024-03-25T16:15:14.357 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-25T16:15:14.469 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 15 MB/s | 21 MB 00:01 2024-03-25T16:15:19.259 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 16 kB/s | 1.7 kB 00:00 2024-03-25T16:15:19.541 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T16:15:21.397 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:15:21.397 INFO:teuthology.orchestra.run.smithi183.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:15:21.506 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:15:21.517 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:15:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:15:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:Install 4 Packages 2024-03-25T16:15:21.518 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:15:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 76 M 2024-03-25T16:15:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 123 M 2024-03-25T16:15:21.518 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:15:21.773 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-25T16:15:21.774 INFO:teuthology.orchestra.run.smithi007.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-25T16:15:21.833 INFO:teuthology.orchestra.run.smithi183.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 190 kB/s | 44 kB 00:00 2024-03-25T16:15:21.869 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-25T16:15:21.877 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout:Install 4 Packages 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 76 M 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 123 M 2024-03-25T16:15:21.878 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:15:22.226 INFO:teuthology.orchestra.run.smithi007.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 146 kB/s | 44 kB 00:00 2024-03-25T16:15:24.100 INFO:teuthology.orchestra.run.smithi183.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 6.0 MB/s | 15 MB 00:02 2024-03-25T16:15:25.008 INFO:teuthology.orchestra.run.smithi183.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 9.8 MB/s | 33 MB 00:03 2024-03-25T16:15:25.392 INFO:teuthology.orchestra.run.smithi007.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 4.3 MB/s | 15 MB 00:03 2024-03-25T16:15:26.242 INFO:teuthology.orchestra.run.smithi007.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 7.7 MB/s | 33 MB 00:04 2024-03-25T16:15:26.516 INFO:teuthology.orchestra.run.smithi183.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.9 MB/s | 27 MB 00:04 2024-03-25T16:15:26.520 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:26.520 INFO:teuthology.orchestra.run.smithi183.stdout:Total 15 MB/s | 76 MB 00:05 2024-03-25T16:15:27.118 INFO:teuthology.orchestra.run.smithi007.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 5.6 MB/s | 27 MB 00:04 2024-03-25T16:15:27.121 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:15:27.121 INFO:teuthology.orchestra.run.smithi007.stdout:Total 14 MB/s | 76 MB 00:05 2024-03-25T16:15:27.184 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:15:27.550 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:15:27.582 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:15:27.816 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:15:28.220 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:15:28.220 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:15:28.566 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:15:28.567 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:15:29.291 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:15:31.030 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:15:31.639 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:15:31.733 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:31.923 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:15:32.567 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:15:32.662 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:34.472 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:34.602 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:35.560 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:15:35.699 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:39.284 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:39.394 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:40.769 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:15:40.808 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:44.117 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:15:45.909 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:13.640 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:13.640 INFO:teuthology.orchestra.run.smithi183.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:16:13.640 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:16:13.686 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:14.043 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:14.043 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:16:14.043 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:14.043 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:14.475 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:16:14.475 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:16:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:16:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:14.476 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:16:14.476 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:16:14.802 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:14.905 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:14.906 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:14.906 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:16:14.906 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:16:14.906 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:16:14.906 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-25T16:16:14.906 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi183.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:16:14.906 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:14.987 INFO:teuthology.orchestra.run.smithi183.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:14.987 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:14.987 DEBUG:teuthology.orchestra.run.smithi183:> sudo rpm -qi grub2-tools 2024-03-25T16:16:15.021 INFO:teuthology.orchestra.run.smithi183.stdout:Name : grub2-tools 2024-03-25T16:16:15.021 INFO:teuthology.orchestra.run.smithi183.stdout:Epoch : 1 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Version : 2.06 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Release : 68.el9 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Architecture: x86_64 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Group : Unspecified 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Size : 8264254 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:License : GPLv3+ 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Packager : builder@centos.org 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:Vendor : CentOS 2024-03-25T16:16:15.022 INFO:teuthology.orchestra.run.smithi183.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout:Summary : Support tools for GRUB. 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout:Description : 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout:hardware devices. 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:16:15.023 INFO:teuthology.orchestra.run.smithi183.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:16:15.024 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:16:15.024 INFO:teuthology.task.kernel:Updating grub on smithi183 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:16:15.024 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:16:15.647 INFO:teuthology.orchestra.run.smithi183.stderr:Generating grub configuration file ... 2024-03-25T16:16:16.776 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:16.776 INFO:teuthology.orchestra.run.smithi007.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-25T16:16:16.777 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:16:16.822 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:16.974 INFO:teuthology.orchestra.run.smithi183.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:16:17.015 INFO:teuthology.orchestra.run.smithi183.stderr:done 2024-03-25T16:16:17.018 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-25T16:16:17.032 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.rG20k9FBOB 2024-03-25T16:16:17.032 DEBUG:teuthology.orchestra.run.smithi183:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.rG20k9FBOB 2024-03-25T16:16:17.100 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.rG20k9FBOB 2024-03-25T16:16:17.177 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:17.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-25T16:16:17.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-25T16:16:17.177 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-25T16:16:17.250 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.rG20k9FBOB is 6KB 2024-03-25T16:16:17.301 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.rG20k9FBOB 2024-03-25T16:16:17.316 DEBUG:teuthology.orchestra.run.smithi183:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:16:17.380 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:16:17.380 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:16:17.380 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:16:17.381 INFO:teuthology.orchestra.run.smithi183.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:16:17.382 DEBUG:teuthology.orchestra.run.smithi183:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:16:17.532 DEBUG:teuthology.orchestra.run.smithi183:> sudo shutdown -r now 2024-03-25T16:16:17.620 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:16:17.621 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:16:17.947 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:18.058 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.059 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.059 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.059 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-25T16:16:18.059 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-25T16:16:18.059 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-25T16:16:18.059 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi007.front.sepia.ceph.com, path=None, version=distro) 2024-03-25T16:16:18.059 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q kernel | sort -rV | head -n 1 2024-03-25T16:16:18.142 INFO:teuthology.orchestra.run.smithi007.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-25T16:16:18.143 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.143 DEBUG:teuthology.orchestra.run.smithi007:> sudo rpm -qi grub2-tools 2024-03-25T16:16:18.183 INFO:teuthology.orchestra.run.smithi007.stdout:Name : grub2-tools 2024-03-25T16:16:18.183 INFO:teuthology.orchestra.run.smithi007.stdout:Epoch : 1 2024-03-25T16:16:18.183 INFO:teuthology.orchestra.run.smithi007.stdout:Version : 2.06 2024-03-25T16:16:18.183 INFO:teuthology.orchestra.run.smithi007.stdout:Release : 68.el9 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Architecture: x86_64 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Group : Unspecified 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Size : 8264254 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:License : GPLv3+ 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-25T16:16:18.184 INFO:teuthology.orchestra.run.smithi007.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:Packager : builder@centos.org 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:Vendor : CentOS 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:Summary : Support tools for GRUB. 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:Description : 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-25T16:16:18.185 INFO:teuthology.orchestra.run.smithi007.stdout:hardware devices. 2024-03-25T16:16:18.186 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:16:18.186 INFO:teuthology.orchestra.run.smithi007.stdout:This subpackage provides tools for support of all platforms. 2024-03-25T16:16:18.187 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-25T16:16:18.187 INFO:teuthology.task.kernel:Updating grub on smithi007 to boot 5.14.0-430.el9.x86_64 2024-03-25T16:16:18.187 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-25T16:16:18.768 INFO:teuthology.orchestra.run.smithi007.stderr:Generating grub configuration file ... 2024-03-25T16:16:20.154 INFO:teuthology.orchestra.run.smithi007.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-25T16:16:20.185 INFO:teuthology.orchestra.run.smithi007.stderr:done 2024-03-25T16:16:20.187 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-25T16:16:20.202 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.6VOnOhNj0C 2024-03-25T16:16:20.203 DEBUG:teuthology.orchestra.run.smithi007:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.6VOnOhNj0C 2024-03-25T16:16:20.274 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.6VOnOhNj0C 2024-03-25T16:16:20.425 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.6VOnOhNj0C is 6KB 2024-03-25T16:16:20.477 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.6VOnOhNj0C 2024-03-25T16:16:20.492 DEBUG:teuthology.orchestra.run.smithi007:> sudo /bin/ls /boot/loader/entries || true 2024-03-25T16:16:20.561 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-25T16:16:20.562 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-25T16:16:20.562 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-25T16:16:20.562 INFO:teuthology.orchestra.run.smithi007.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-25T16:16:20.563 DEBUG:teuthology.orchestra.run.smithi007:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-25T16:16:20.731 DEBUG:teuthology.orchestra.run.smithi007:> sudo shutdown -r now 2024-03-25T16:16:47.558 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-25T16:16:47.559 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-25T16:16:47.560 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:16:50.734 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-25T16:16:50.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-25T16:16:50.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:06.079 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T16:17:09.151 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2024-03-25T16:17:15.086 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-25T16:17:15.087 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:18.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.183 2024-03-25T16:17:18.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-25T16:17:18.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:21.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.7 2024-03-25T16:17:30.150 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-25T16:17:30.151 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:17:33.218 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-25T16:17:33.219 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:18:02.296 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-25T16:18:02.771 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-25T16:18:02.771 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:18:02.772 DEBUG:teuthology.orchestra.run.smithi183:> uname -r 2024-03-25T16:18:02.797 INFO:teuthology.orchestra.run.smithi183.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:18:02.798 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:18:02.798 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:18:02.798 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:18:03.798 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-25T16:18:03.799 DEBUG:teuthology.orchestra.run.smithi183:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:18:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:ttyS1 2024-03-25T16:18:03.904 DEBUG:teuthology.parallel:result is None 2024-03-25T16:18:04.994 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-25T16:18:05.746 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-25T16:18:05.746 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-430.el9.x86_64"... 2024-03-25T16:18:05.747 DEBUG:teuthology.orchestra.run.smithi007:> uname -r 2024-03-25T16:18:05.760 INFO:teuthology.orchestra.run.smithi007.stdout:5.14.0-430.el9.x86_64 2024-03-25T16:18:05.760 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-25T16:18:05.761 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-25T16:18:05.761 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-25T16:18:06.761 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-25T16:18:06.762 DEBUG:teuthology.orchestra.run.smithi007:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-25T16:18:06.817 INFO:teuthology.orchestra.run.smithi007.stdout:ttyS1 2024-03-25T16:18:06.850 DEBUG:teuthology.parallel:result is None 2024-03-25T16:18:06.850 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-25T16:18:06.858 INFO:teuthology.task.internal:Creating test directory... 2024-03-25T16:18:06.859 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:18:06.861 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-25T16:18:06.880 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-25T16:18:06.897 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-25T16:18:06.929 INFO:teuthology.task.internal:Creating archive directory... 2024-03-25T16:18:06.930 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:18:06.932 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-25T16:18:07.081 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-25T16:18:07.088 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-25T16:18:07.089 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:18:07.091 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-25T16:18:07.123 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.136 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.148 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.162 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-25T16:18:07.164 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-25T16:18:07.171 INFO:teuthology.task.internal:Configuring sudo... 2024-03-25T16:18:07.171 DEBUG:teuthology.orchestra.run.smithi007:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:18:07.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-25T16:18:07.213 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-25T16:18:07.223 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-25T16:18:07.224 DEBUG:teuthology.orchestra.run.smithi007:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:18:07.255 DEBUG:teuthology.orchestra.run.smithi183:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-25T16:18:07.271 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:07.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:07.412 DEBUG:teuthology.orchestra.run.smithi007:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:07.496 DEBUG:teuthology.orchestra.run.smithi007:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:07.541 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T16:18:07.541 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:18:07.611 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:07.654 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-25T16:18:07.704 DEBUG:teuthology.orchestra.run.smithi183:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:07.786 DEBUG:teuthology.orchestra.run.smithi183:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-25T16:18:07.832 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T16:18:07.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-25T16:18:07.901 DEBUG:teuthology.orchestra.run.smithi007:> sudo service rsyslog restart 2024-03-25T16:18:07.903 DEBUG:teuthology.orchestra.run.smithi183:> sudo service rsyslog restart 2024-03-25T16:18:07.954 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:18:07.988 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T16:18:08.502 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-25T16:18:08.510 INFO:teuthology.task.internal:Starting timer... 2024-03-25T16:18:08.510 INFO:teuthology.run_tasks:Running task pcp... 2024-03-25T16:18:08.557 INFO:teuthology.run_tasks:Running task selinux... 2024-03-25T16:18:08.589 DEBUG:teuthology.orchestra.run.smithi007:> sudo service auditd rotate 2024-03-25T16:18:08.641 INFO:teuthology.orchestra.run.smithi007.stdout:Rotating logs: 2024-03-25T16:18:08.643 DEBUG:teuthology.orchestra.run.smithi183:> sudo service auditd rotate 2024-03-25T16:18:08.689 INFO:teuthology.orchestra.run.smithi183.stdout:Rotating logs: 2024-03-25T16:18:08.691 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-25T16:18:08.691 DEBUG:teuthology.orchestra.run.smithi007:> /usr/sbin/getenforce 2024-03-25T16:18:08.720 INFO:teuthology.orchestra.run.smithi007.stdout:Permissive 2024-03-25T16:18:08.720 DEBUG:teuthology.orchestra.run.smithi183:> /usr/sbin/getenforce 2024-03-25T16:18:08.755 INFO:teuthology.orchestra.run.smithi183.stdout:Permissive 2024-03-25T16:18:08.756 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi007.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi183.front.sepia.ceph.com': 'permissive'} 2024-03-25T16:18:08.756 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:18:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:18:08.792 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T16:18:08.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T16:18:08.824 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-25T16:18:08.824 DEBUG:teuthology.orchestra.run.smithi007:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:18:08.871 DEBUG:teuthology.orchestra.run.smithi183:> sudo /usr/sbin/setenforce permissive 2024-03-25T16:18:08.908 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-25T16:18:08.917 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-25T16:18:08.936 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-25T16:18:08.937 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi007.front.sepia.ceph.com,smithi183.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-25T16:22:23.660 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi007.front.sepia.ceph.com'), Remote(name='ubuntu@smithi183.front.sepia.ceph.com')] 2024-03-25T16:22:23.662 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-25T16:22:23.663 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi007.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:23.758 DEBUG:teuthology.orchestra.run.smithi007:> true 2024-03-25T16:22:23.836 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi007.front.sepia.ceph.com' 2024-03-25T16:22:23.837 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-25T16:22:23.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi183.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-25T16:22:23.926 DEBUG:teuthology.orchestra.run.smithi183:> true 2024-03-25T16:22:24.002 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi183.front.sepia.ceph.com' 2024-03-25T16:22:24.004 INFO:teuthology.run_tasks:Running task clock... 2024-03-25T16:22:24.018 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-25T16:22:24.019 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:22:24.019 DEBUG:teuthology.orchestra.run.smithi007:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:22:24.021 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-25T16:22:24.021 DEBUG:teuthology.orchestra.run.smithi183:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T16:22:24.051 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:22:24.067 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:22:24.074 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-25T16:22:24.090 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-25T16:22:24.097 INFO:teuthology.orchestra.run.smithi007.stderr:sudo: ntpd: command not found 2024-03-25T16:22:24.111 INFO:teuthology.orchestra.run.smithi007.stdout:506 Cannot talk to daemon 2024-03-25T16:22:24.120 INFO:teuthology.orchestra.run.smithi183.stderr:sudo: ntpd: command not found 2024-03-25T16:22:24.127 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:22:24.133 INFO:teuthology.orchestra.run.smithi183.stdout:506 Cannot talk to daemon 2024-03-25T16:22:24.143 INFO:teuthology.orchestra.run.smithi007.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:22:24.150 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-25T16:22:24.167 INFO:teuthology.orchestra.run.smithi183.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-25T16:22:24.198 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:22:24.201 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:22:24.202 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-25T16:22:24.202 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.202 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.202 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.202 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.217 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-25T16:22:24.220 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T16:22:24.220 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-25T16:22:24.221 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.221 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.221 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.221 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T16:22:24.222 INFO:teuthology.run_tasks:Running task install... 2024-03-25T16:22:24.232 DEBUG:teuthology.task.install:project ceph 2024-03-25T16:22:24.232 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-25T16:22:24.233 DEBUG:teuthology.task.install:config {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-25T16:22:24.233 INFO:teuthology.task.install:Using flavor: default 2024-03-25T16:22:24.239 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse', 'ceph-common-dbg', 'ceph-fuse-dbg', 'ceph-mds-dbg', 'ceph-mgr-dbg', 'ceph-mon-dbg', 'ceph-osd-dbg', 'ceph-test-dbg', 'libcephfs2-dbg', 'librados2-dbg', 'libradosstriper1-dbg', 'librbd1-dbg', 'librgw2-dbg', 'radosgw-dbg', 'rbd-fuse-dbg', 'rbd-mirror-dbg', 'rbd-nbd-dbg'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd', 'ceph-base-debuginfo', 'ceph-common-debuginfo', 'ceph-immutable-object-cache-debuginfo', 'ceph-radosgw-debuginfo', 'ceph-test-debuginfo', 'ceph-base-debuginfo', 'ceph-mgr-debuginfo', 'ceph-mds-debuginfo', 'ceph-mon-debuginfo', 'ceph-osd-debuginfo', 'ceph-fuse-debuginfo', 'librados-devel-debuginfo', 'libcephfs2-debuginfo', 'librados2-debuginfo', 'librbd1-debuginfo', 'python3-cephfs-debuginfo', 'python3-rados-debuginfo', 'python3-rbd-debuginfo', 'python3-rgw-debuginfo', 'rbd-fuse-debuginfo', 'rbd-mirror-debuginfo', 'rbd-nbd-debuginfo']} 2024-03-25T16:22:24.239 INFO:teuthology.task.install:extra packages: [] 2024-03-25T16:22:24.239 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:22:24.240 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:22:24.242 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': True, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-25T16:22:24.242 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T16:22:24.383 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:22:24.383 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:22:24.387 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-25T16:22:24.387 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-25T16:22:24.852 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:22:24.852 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T16:22:24.852 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:22:24.880 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:22:24.880 DEBUG:teuthology.orchestra.run.smithi183:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:22:24.948 DEBUG:teuthology.orchestra.run.smithi183:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:22:25.028 DEBUG:teuthology.orchestra.run.smithi183:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:22:25.094 INFO:teuthology.orchestra.run.smithi183.stdout:check_obsoletes = 1 2024-03-25T16:22:25.095 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum clean all 2024-03-25T16:22:25.199 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-25T16:22:25.199 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T16:22:25.199 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-25T16:22:25.232 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, ceph-base-debuginfo, ceph-common-debuginfo, ceph-immutable-object-cache-debuginfo, ceph-radosgw-debuginfo, ceph-test-debuginfo, ceph-base-debuginfo, ceph-mgr-debuginfo, ceph-mds-debuginfo, ceph-mon-debuginfo, ceph-osd-debuginfo, ceph-fuse-debuginfo, librados-devel-debuginfo, libcephfs2-debuginfo, librados2-debuginfo, librbd1-debuginfo, python3-cephfs-debuginfo, python3-rados-debuginfo, python3-rbd-debuginfo, python3-rgw-debuginfo, rbd-fuse-debuginfo, rbd-mirror-debuginfo, rbd-nbd-debuginfo on remote rpm x86_64 2024-03-25T16:22:25.232 DEBUG:teuthology.orchestra.run.smithi007:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-25T16:22:25.303 DEBUG:teuthology.orchestra.run.smithi007:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-25T16:22:25.336 INFO:teuthology.orchestra.run.smithi183.stdout:74 files removed 2024-03-25T16:22:25.363 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw 2024-03-25T16:22:25.387 DEBUG:teuthology.orchestra.run.smithi007:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-25T16:22:25.455 INFO:teuthology.orchestra.run.smithi007.stdout:check_obsoletes = 1 2024-03-25T16:22:25.456 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum clean all 2024-03-25T16:22:25.723 INFO:teuthology.orchestra.run.smithi007.stdout:74 files removed 2024-03-25T16:22:25.749 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw 2024-03-25T16:22:26.633 INFO:teuthology.orchestra.run.smithi183.stdout:ceph packages for x86_64 85 kB/s | 82 kB 00:00 2024-03-25T16:22:27.074 INFO:teuthology.orchestra.run.smithi007.stdout:ceph packages for x86_64 80 kB/s | 82 kB 00:01 2024-03-25T16:22:27.466 INFO:teuthology.orchestra.run.smithi183.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:22:27.908 INFO:teuthology.orchestra.run.smithi007.stdout:ceph noarch packages 13 kB/s | 10 kB 00:00 2024-03-25T16:22:28.325 INFO:teuthology.orchestra.run.smithi183.stdout:ceph source packages 2.3 kB/s | 1.9 kB 00:00 2024-03-25T16:22:28.734 INFO:teuthology.orchestra.run.smithi007.stdout:ceph source packages 2.4 kB/s | 1.9 kB 00:00 2024-03-25T16:22:28.737 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-25T16:22:29.352 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-25T16:22:30.111 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-25T16:22:30.894 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS-9 - BaseOS 5.3 MB/s | 2.0 MB 00:00 2024-03-25T16:22:32.152 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - BaseOS 4.7 MB/s | 8.0 MB 00:01 2024-03-25T16:22:32.243 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - BaseOS 8.1 MB/s | 8.0 MB 00:00 2024-03-25T16:22:33.753 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - AppStream 33 MB/s | 19 MB 00:00 2024-03-25T16:22:33.944 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - AppStream 32 MB/s | 19 MB 00:00 2024-03-25T16:22:38.248 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.1 MB 00:00 2024-03-25T16:22:38.945 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - CRB 3.6 MB/s | 6.1 MB 00:01 2024-03-25T16:22:39.687 INFO:teuthology.orchestra.run.smithi007.stdout:CentOS Stream 9 - Extras packages 63 kB/s | 16 kB 00:00 2024-03-25T16:22:40.321 INFO:teuthology.orchestra.run.smithi183.stdout:CentOS Stream 9 - Extras packages 82 kB/s | 16 kB 00:00 2024-03-25T16:22:41.030 INFO:teuthology.orchestra.run.smithi007.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-03-25T16:22:41.705 INFO:teuthology.orchestra.run.smithi183.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-03-25T16:22:46.097 INFO:teuthology.orchestra.run.smithi007.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-25T16:22:46.672 INFO:teuthology.orchestra.run.smithi183.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-03-25T16:22:48.612 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrading: 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:22:48.616 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:22:48.617 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:22:48.618 INFO:teuthology.orchestra.run.smithi007.stdout:Installing weak dependencies: 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout:Install 30 Packages 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout:Upgrade 2 Packages 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 58 M 2024-03-25T16:22:48.619 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:22:49.125 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:22:49.129 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:22:49.129 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:22:49.129 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:22:49.129 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:22:49.129 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-25T16:22:49.129 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrading: 2024-03-25T16:22:49.129 INFO:teuthology.orchestra.run.smithi183.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-25T16:22:49.130 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-25T16:22:49.131 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout:Installing weak dependencies: 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout:Install 30 Packages 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout:Upgrade 2 Packages 2024-03-25T16:22:49.132 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:22:49.133 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 58 M 2024-03-25T16:22:49.133 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:22:53.614 INFO:teuthology.orchestra.run.smithi007.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 5.3 MB 00:04 2024-03-25T16:22:53.698 INFO:teuthology.orchestra.run.smithi007.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 303 kB/s | 25 kB 00:00 2024-03-25T16:22:54.348 INFO:teuthology.orchestra.run.smithi007.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 742 kB 00:00 2024-03-25T16:22:54.540 INFO:teuthology.orchestra.run.smithi007.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.7 MB/s | 9.7 MB 00:05 2024-03-25T16:22:54.865 INFO:teuthology.orchestra.run.smithi007.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 964 kB/s | 498 kB 00:00 2024-03-25T16:22:54.948 INFO:teuthology.orchestra.run.smithi007.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 541 kB/s | 45 kB 00:00 2024-03-25T16:22:55.115 INFO:teuthology.orchestra.run.smithi007.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 788 kB/s | 131 kB 00:00 2024-03-25T16:22:55.366 INFO:teuthology.orchestra.run.smithi007.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 648 kB/s | 162 kB 00:00 2024-03-25T16:22:55.707 INFO:teuthology.orchestra.run.smithi007.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 938 kB/s | 320 kB 00:00 2024-03-25T16:22:56.024 INFO:teuthology.orchestra.run.smithi007.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 940 kB/s | 298 kB 00:00 2024-03-25T16:22:56.208 INFO:teuthology.orchestra.run.smithi007.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 541 kB/s | 99 kB 00:00 2024-03-25T16:22:56.317 INFO:teuthology.orchestra.run.smithi007.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 977 kB/s | 106 kB 00:00 2024-03-25T16:22:56.359 INFO:teuthology.orchestra.run.smithi007.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-25T16:22:56.392 INFO:teuthology.orchestra.run.smithi007.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-25T16:22:56.451 INFO:teuthology.orchestra.run.smithi007.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-25T16:22:56.493 INFO:teuthology.orchestra.run.smithi007.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-25T16:22:56.527 INFO:teuthology.orchestra.run.smithi007.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-25T16:22:56.743 INFO:teuthology.orchestra.run.smithi007.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 1.0 MB/s | 216 kB 00:00 2024-03-25T16:22:56.802 INFO:teuthology.orchestra.run.smithi007.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 597 kB/s | 35 kB 00:00 2024-03-25T16:22:56.861 INFO:teuthology.orchestra.run.smithi007.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 763 kB/s | 44 kB 00:00 2024-03-25T16:22:57.328 INFO:teuthology.orchestra.run.smithi007.stdout:(21/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 5.1 MB 00:02 2024-03-25T16:22:57.370 INFO:teuthology.orchestra.run.smithi007.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 607 kB/s | 309 kB 00:00 2024-03-25T16:22:57.520 INFO:teuthology.orchestra.run.smithi007.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 173 kB/s | 26 kB 00:00 2024-03-25T16:22:57.653 INFO:teuthology.orchestra.run.smithi007.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 364 kB/s | 49 kB 00:00 2024-03-25T16:22:57.737 INFO:teuthology.orchestra.run.smithi007.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 809 kB/s | 67 kB 00:00 2024-03-25T16:22:57.987 INFO:teuthology.orchestra.run.smithi007.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 3.3 MB/s | 838 kB 00:00 2024-03-25T16:22:58.087 INFO:teuthology.orchestra.run.smithi007.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-25T16:22:58.296 INFO:teuthology.orchestra.run.smithi007.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 7.6 MB/s | 1.6 MB 00:00 2024-03-25T16:22:58.363 INFO:teuthology.orchestra.run.smithi007.stdout:(29/32): libarrow-9.0.0-10.el9.x86_64.rpm 4.3 MB/s | 4.4 MB 00:01 2024-03-25T16:23:00.088 INFO:teuthology.orchestra.run.smithi183.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 514 kB/s | 5.3 MB 00:10 2024-03-25T16:23:00.571 INFO:teuthology.orchestra.run.smithi183.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 52 kB/s | 25 kB 00:00 2024-03-25T16:23:00.921 INFO:teuthology.orchestra.run.smithi007.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 3.1 MB 00:02 2024-03-25T16:23:01.905 INFO:teuthology.orchestra.run.smithi183.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 556 kB/s | 742 kB 00:01 2024-03-25T16:23:02.088 INFO:teuthology.orchestra.run.smithi183.stdout:(4/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 789 kB/s | 9.7 MB 00:12 2024-03-25T16:23:02.396 INFO:teuthology.orchestra.run.smithi007.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 843 kB/s | 3.4 MB 00:04 2024-03-25T16:23:02.782 INFO:teuthology.orchestra.run.smithi183.stdout:(5/32): libradosstriper1-19.0.0-1439.g5dcb190d. 570 kB/s | 498 kB 00:00 2024-03-25T16:23:02.888 INFO:teuthology.orchestra.run.smithi183.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 416 kB/s | 45 kB 00:00 2024-03-25T16:23:03.147 INFO:teuthology.orchestra.run.smithi183.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 508 kB/s | 131 kB 00:00 2024-03-25T16:23:03.414 INFO:teuthology.orchestra.run.smithi183.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 607 kB/s | 162 kB 00:00 2024-03-25T16:23:04.139 INFO:teuthology.orchestra.run.smithi183.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 442 kB/s | 320 kB 00:00 2024-03-25T16:23:04.764 INFO:teuthology.orchestra.run.smithi183.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 477 kB/s | 298 kB 00:00 2024-03-25T16:23:05.014 INFO:teuthology.orchestra.run.smithi183.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 397 kB/s | 99 kB 00:00 2024-03-25T16:23:05.139 INFO:teuthology.orchestra.run.smithi183.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 848 kB/s | 106 kB 00:00 2024-03-25T16:23:05.190 INFO:teuthology.orchestra.run.smithi183.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-25T16:23:05.232 INFO:teuthology.orchestra.run.smithi183.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-25T16:23:05.307 INFO:teuthology.orchestra.run.smithi183.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-25T16:23:05.349 INFO:teuthology.orchestra.run.smithi183.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.7 MB/s | 282 kB 00:00 2024-03-25T16:23:05.391 INFO:teuthology.orchestra.run.smithi183.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 994 kB/s | 41 kB 00:00 2024-03-25T16:23:05.733 INFO:teuthology.orchestra.run.smithi183.stdout:(18/32): qatlib-23.11.0-1.el9.x86_64.rpm 634 kB/s | 216 kB 00:00 2024-03-25T16:23:05.967 INFO:teuthology.orchestra.run.smithi183.stdout:(19/32): qatlib-service-23.11.0-1.el9.x86_64.rp 150 kB/s | 35 kB 00:00 2024-03-25T16:23:06.217 INFO:teuthology.orchestra.run.smithi183.stdout:(20/32): qatzip-libs-1.1.2-2.el9.x86_64.rpm 178 kB/s | 44 kB 00:00 2024-03-25T16:23:06.363 INFO:teuthology.orchestra.run.smithi007.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 20 MB 00:17 2024-03-25T16:23:06.368 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:06.369 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.3 MB/s | 58 MB 00:17 2024-03-25T16:23:06.404 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:23:06.446 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:23:06.446 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:23:06.450 INFO:teuthology.orchestra.run.smithi183.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.3 MB/s | 309 kB 00:00 2024-03-25T16:23:06.701 INFO:teuthology.orchestra.run.smithi183.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-03-25T16:23:06.759 INFO:teuthology.orchestra.run.smithi183.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 447 kB/s | 26 kB 00:00 2024-03-25T16:23:06.809 INFO:teuthology.orchestra.run.smithi183.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 972 kB/s | 49 kB 00:00 2024-03-25T16:23:06.860 INFO:teuthology.orchestra.run.smithi183.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-03-25T16:23:06.927 INFO:teuthology.orchestra.run.smithi183.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 12 MB/s | 838 kB 00:00 2024-03-25T16:23:06.939 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:23:06.939 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:23:06.985 INFO:teuthology.orchestra.run.smithi183.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-03-25T16:23:07.069 INFO:teuthology.orchestra.run.smithi183.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 19 MB/s | 1.6 MB 00:00 2024-03-25T16:23:07.936 INFO:teuthology.orchestra.run.smithi183.stdout:(29/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 893 kB/s | 5.1 MB 00:05 2024-03-25T16:23:08.437 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:23:08.525 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:23:08.559 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:23:08.585 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:23:08.607 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:08.624 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:08.791 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:08.846 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:09.181 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:23:09.228 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:09.325 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:09.418 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:23:09.446 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:09.508 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:09.537 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:23:09.581 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:23:09.651 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:09.690 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:09.760 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:09.779 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:23:09.890 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:23:09.930 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:23:09.971 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:23:09.995 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:23:10.024 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:23:10.057 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:23:10.387 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:23:10.411 INFO:teuthology.orchestra.run.smithi007.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:10.485 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:10.516 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:23:10.556 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:23:10.661 INFO:teuthology.orchestra.run.smithi183.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 3.1 MB 00:02 2024-03-25T16:23:11.031 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:23:11.064 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:11.136 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:11.206 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:23:11.241 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:23:12.702 INFO:teuthology.orchestra.run.smithi183.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 613 kB/s | 3.4 MB 00:05 2024-03-25T16:23:13.453 INFO:teuthology.orchestra.run.smithi183.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 848 kB/s | 20 MB 00:24 2024-03-25T16:23:13.456 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:23:13.456 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.4 MB/s | 58 MB 00:24 2024-03-25T16:23:13.486 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:23:13.521 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:23:13.521 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:23:13.559 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:13.596 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:13.988 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:23:13.988 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:23:14.220 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:14.258 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:14.353 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:14.353 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:23:14.353 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:23:14.354 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:23:14.384 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:15.442 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:23:15.535 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-25T16:23:15.565 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-25T16:23:15.591 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-25T16:23:15.622 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:15.638 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 4/34 2024-03-25T16:23:15.814 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:15.869 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 5/34 2024-03-25T16:23:16.189 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-25T16:23:16.259 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:16.350 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:16.438 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-25T16:23:16.460 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:16.523 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-25T16:23:16.552 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-25T16:23:16.587 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-25T16:23:16.655 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:16.687 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:16.750 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-25T16:23:16.794 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 14/34 2024-03-25T16:23:16.932 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-25T16:23:16.971 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-25T16:23:17.019 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-25T16:23:17.051 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-25T16:23:17.080 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-25T16:23:17.113 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-25T16:23:17.438 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-25T16:23:17.468 INFO:teuthology.orchestra.run.smithi183.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:17.557 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-25T16:23:17.589 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-25T16:23:17.635 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-25T16:23:18.093 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-25T16:23:18.120 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:18.191 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-25T16:23:18.259 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-25T16:23:18.289 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-25T16:23:20.418 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:20.460 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:21.008 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-25T16:23:21.113 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:21.207 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-25T16:23:21.207 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-25T16:23:21.207 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-25T16:23:21.208 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:23:21.248 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:24.170 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:24.170 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /sys 2024-03-25T16:23:24.170 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /proc 2024-03-25T16:23:24.171 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /mnt 2024-03-25T16:23:24.171 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /var/tmp 2024-03-25T16:23:24.171 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /home 2024-03-25T16:23:24.171 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /root 2024-03-25T16:23:24.171 INFO:teuthology.orchestra.run.smithi007.stdout:skipping the directory /tmp 2024-03-25T16:23:24.171 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:23:25.084 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:25.108 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:25.108 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:25.108 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:23:25.108 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:25.108 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:25.108 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:23:25.110 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:25.194 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:25.195 INFO:teuthology.orchestra.run.smithi007.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:26.397 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:23:26.398 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:23:26.399 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:23:26.399 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:23:26.399 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:23:26.399 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:23:26.399 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:23:26.399 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:23:26.400 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:23:26.400 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:23:26.400 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:23:26.400 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:26.400 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:23:26.400 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout:Upgraded: 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:23:27.393 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.394 INFO:teuthology.orchestra.run.smithi007.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:23:27.395 INFO:teuthology.orchestra.run.smithi007.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:23:27.395 INFO:teuthology.orchestra.run.smithi007.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:23:27.395 INFO:teuthology.orchestra.run.smithi007.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.395 INFO:teuthology.orchestra.run.smithi007.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:23:27.395 INFO:teuthology.orchestra.run.smithi007.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.395 INFO:teuthology.orchestra.run.smithi007.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:23:27.395 INFO:teuthology.orchestra.run.smithi007.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:23:27.396 INFO:teuthology.orchestra.run.smithi007.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:23:27.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:23:27.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.396 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:23:27.397 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:23:27.646 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test 2024-03-25T16:23:28.216 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:00:42 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:23:28.370 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:23:28.371 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:23:28.372 INFO:teuthology.orchestra.run.smithi007.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:23:28.372 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:23:28.372 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:23:28.372 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:23:28.372 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:23:28.372 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-03-25T16:23:28.372 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:23:28.373 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 48 M 2024-03-25T16:23:28.373 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 204 M 2024-03-25T16:23:28.373 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:23:28.812 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-25T16:23:28.862 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-25T16:23:28.896 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:23:29.029 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 1.4 MB/s | 187 kB 00:00 2024-03-25T16:23:29.471 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 199 kB/s | 163 kB 00:00 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /sys 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /proc 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /mnt 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /var/tmp 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /home 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /root 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout:skipping the directory /tmp 2024-03-25T16:23:30.351 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:23:31.277 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:31.301 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-25T16:23:31.301 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:23:31.301 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-25T16:23:31.301 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:31.301 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-25T16:23:31.301 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:23:31.303 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:31.396 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-25T16:23:31.396 INFO:teuthology.orchestra.run.smithi183.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-25T16:23:32.568 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 20/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 21/34 2024-03-25T16:23:32.569 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 22/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-25T16:23:32.570 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-25T16:23:33.714 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout:Upgraded: 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-25T16:23:33.715 INFO:teuthology.orchestra.run.smithi183.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.716 INFO:teuthology.orchestra.run.smithi183.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: re2-1:20211101-3.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:23:33.717 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:23:33.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test 2024-03-25T16:23:34.554 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:00:48 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:23:34.706 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:23:34.706 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:23:34.706 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-25T16:23:34.707 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:23:34.708 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 48 M 2024-03-25T16:23:34.708 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 204 M 2024-03-25T16:23:34.708 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:23:35.135 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-25T16:23:35.210 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.6 MB/s | 300 kB 00:00 2024-03-25T16:23:35.244 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-25T16:23:35.444 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): jq-1.6-16.el9.x86_64.rpm 937 kB/s | 187 kB 00:00 2024-03-25T16:23:35.878 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 183 kB/s | 163 kB 00:00 2024-03-25T16:24:40.401 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 675 kB/s | 47 MB 01:11 2024-03-25T16:24:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:24:40.403 INFO:teuthology.orchestra.run.smithi007.stdout:Total 685 kB/s | 48 MB 01:12 2024-03-25T16:24:40.415 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:24:40.435 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:24:40.436 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:24:40.833 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:24:40.834 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:24:41.435 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:24:41.513 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:24:41.550 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:24:41.598 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:24:41.646 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:24:41.674 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:24:46.388 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:24:46.469 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:24:48.760 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:24:48.760 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:24:48.760 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:24:48.760 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:24:48.760 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:24:48.760 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:24:49.125 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:24:49.125 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:24:49.125 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:24:49.125 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:49.125 INFO:teuthology.orchestra.run.smithi007.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:24:49.126 INFO:teuthology.orchestra.run.smithi007.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:24:49.126 INFO:teuthology.orchestra.run.smithi007.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:24:49.126 INFO:teuthology.orchestra.run.smithi007.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:24:49.126 INFO:teuthology.orchestra.run.smithi007.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:24:49.126 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:24:49.126 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:24:49.450 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph 2024-03-25T16:24:50.021 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:04 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:24:50.188 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:24:50.191 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:24:50.192 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:24:50.193 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout:========================================================================================= 2024-03-25T16:24:50.194 INFO:teuthology.orchestra.run.smithi007.stdout:Install 36 Packages 2024-03-25T16:24:50.195 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:24:50.195 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 30 M 2024-03-25T16:24:50.195 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 107 M 2024-03-25T16:24:50.195 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:24:51.083 INFO:teuthology.orchestra.run.smithi007.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 13 kB/s | 6.6 kB 00:00 2024-03-25T16:24:54.658 INFO:teuthology.orchestra.run.smithi007.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 375 kB/s | 1.5 MB 00:04 2024-03-25T16:24:55.224 INFO:teuthology.orchestra.run.smithi007.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 469 kB/s | 2.1 MB 00:04 2024-03-25T16:24:55.924 INFO:teuthology.orchestra.run.smithi007.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 353 kB/s | 247 kB 00:00 2024-03-25T16:24:56.050 INFO:teuthology.orchestra.run.smithi007.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.5 MB/s | 187 kB 00:00 2024-03-25T16:24:56.092 INFO:teuthology.orchestra.run.smithi007.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:24:56.125 INFO:teuthology.orchestra.run.smithi007.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 951 kB/s | 32 kB 00:00 2024-03-25T16:24:56.167 INFO:teuthology.orchestra.run.smithi007.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-25T16:24:56.201 INFO:teuthology.orchestra.run.smithi007.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-25T16:24:56.243 INFO:teuthology.orchestra.run.smithi007.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-25T16:24:56.310 INFO:teuthology.orchestra.run.smithi007.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:24:56.344 INFO:teuthology.orchestra.run.smithi007.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 3.0 MB/s | 103 kB 00:00 2024-03-25T16:24:56.377 INFO:teuthology.orchestra.run.smithi007.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:24:56.411 INFO:teuthology.orchestra.run.smithi007.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 5.5 MB/s | 187 kB 00:00 2024-03-25T16:24:56.694 INFO:teuthology.orchestra.run.smithi007.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 441 kB/s | 125 kB 00:00 2024-03-25T16:24:56.870 INFO:teuthology.orchestra.run.smithi007.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 128 kB/s | 22 kB 00:00 2024-03-25T16:24:57.128 INFO:teuthology.orchestra.run.smithi007.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 585 kB/s | 151 kB 00:00 2024-03-25T16:24:57.187 INFO:teuthology.orchestra.run.smithi007.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 747 kB/s | 43 kB 00:00 2024-03-25T16:24:57.271 INFO:teuthology.orchestra.run.smithi007.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.0 MB/s | 172 kB 00:00 2024-03-25T16:24:57.371 INFO:teuthology.orchestra.run.smithi007.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-25T16:24:57.429 INFO:teuthology.orchestra.run.smithi007.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 183 kB/s | 11 kB 00:00 2024-03-25T16:24:57.496 INFO:teuthology.orchestra.run.smithi007.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 265 kB/s | 18 kB 00:00 2024-03-25T16:24:57.547 INFO:teuthology.orchestra.run.smithi007.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 463 kB/s | 23 kB 00:00 2024-03-25T16:24:57.597 INFO:teuthology.orchestra.run.smithi007.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-25T16:24:57.656 INFO:teuthology.orchestra.run.smithi007.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 339 kB/s | 20 kB 00:00 2024-03-25T16:24:57.723 INFO:teuthology.orchestra.run.smithi007.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 692 kB/s | 46 kB 00:00 2024-03-25T16:24:57.790 INFO:teuthology.orchestra.run.smithi007.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-25T16:24:57.848 INFO:teuthology.orchestra.run.smithi007.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-25T16:24:57.907 INFO:teuthology.orchestra.run.smithi007.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 282 kB/s | 16 kB 00:00 2024-03-25T16:24:57.974 INFO:teuthology.orchestra.run.smithi007.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.3 MB/s | 90 kB 00:00 2024-03-25T16:24:58.041 INFO:teuthology.orchestra.run.smithi007.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 537 kB/s | 36 kB 00:00 2024-03-25T16:24:58.110 INFO:teuthology.orchestra.run.smithi007.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.3 MB/s | 230 kB 00:00 2024-03-25T16:24:58.175 INFO:teuthology.orchestra.run.smithi007.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.4 MB/s | 427 kB 00:00 2024-03-25T16:24:58.242 INFO:teuthology.orchestra.run.smithi007.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 301 kB/s | 20 kB 00:00 2024-03-25T16:25:03.825 INFO:teuthology.orchestra.run.smithi007.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 376 kB/s | 4.7 MB 00:12 2024-03-25T16:25:16.171 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 478 kB/s | 47 MB 01:41 2024-03-25T16:25:16.174 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:16.174 INFO:teuthology.orchestra.run.smithi183.stdout:Total 486 kB/s | 48 MB 01:41 2024-03-25T16:25:16.186 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:25:16.205 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:25:16.205 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:25:16.595 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:25:16.596 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:25:17.189 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:25:17.262 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-25T16:25:17.314 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-25T16:25:17.354 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:25:17.404 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-25T16:25:17.432 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:25:21.932 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-25T16:25:22.015 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:25:24.453 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-25T16:25:24.454 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-25T16:25:24.454 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-25T16:25:24.454 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-25T16:25:24.454 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-25T16:25:24.454 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-25T16:25:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-25T16:25:24.854 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:25:24.854 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: jq-1.6-16.el9.x86_64 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:25:24.855 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:25:25.152 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph 2024-03-25T16:25:25.727 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:02:39 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:25:25.886 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:25:25.888 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-25T16:25:25.888 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:25:25.888 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-25T16:25:25.889 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-25T16:25:25.890 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:25:25.891 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:25:25.892 INFO:teuthology.orchestra.run.smithi183.stdout:========================================================================================= 2024-03-25T16:25:25.892 INFO:teuthology.orchestra.run.smithi183.stdout:Install 36 Packages 2024-03-25T16:25:25.892 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:25:25.892 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 30 M 2024-03-25T16:25:25.892 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 107 M 2024-03-25T16:25:25.892 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:25:26.858 INFO:teuthology.orchestra.run.smithi183.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 15 kB/s | 6.6 kB 00:00 2024-03-25T16:25:30.258 INFO:teuthology.orchestra.run.smithi183.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 396 kB/s | 1.5 MB 00:03 2024-03-25T16:25:32.199 INFO:teuthology.orchestra.run.smithi183.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 375 kB/s | 2.1 MB 00:05 2024-03-25T16:25:33.077 INFO:teuthology.orchestra.run.smithi183.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 283 kB/s | 247 kB 00:00 2024-03-25T16:25:33.191 INFO:teuthology.orchestra.run.smithi183.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 1.6 MB/s | 187 kB 00:00 2024-03-25T16:25:33.233 INFO:teuthology.orchestra.run.smithi183.stdout:(6/36): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-25T16:25:33.258 INFO:teuthology.orchestra.run.smithi183.stdout:(7/36): python3-markupsafe-1.1.1-12.el9.x86_64. 1.2 MB/s | 32 kB 00:00 2024-03-25T16:25:33.284 INFO:teuthology.orchestra.run.smithi183.stdout:(8/36): python3-pytz-2021.1-5.el9.noarch.rpm 1.9 MB/s | 47 kB 00:00 2024-03-25T16:25:33.309 INFO:teuthology.orchestra.run.smithi183.stdout:(9/36): python3-toml-0.10.2-6.el9.noarch.rpm 1.5 MB/s | 39 kB 00:00 2024-03-25T16:25:33.351 INFO:teuthology.orchestra.run.smithi183.stdout:(10/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.7 MB/s | 241 kB 00:00 2024-03-25T16:25:33.418 INFO:teuthology.orchestra.run.smithi183.stdout:(11/36): python3-cryptography-36.0.1-4.el9.x86_ 17 MB/s | 1.1 MB 00:00 2024-03-25T16:25:33.460 INFO:teuthology.orchestra.run.smithi183.stdout:(12/36): python3-ply-3.11-14.el9.noarch.rpm 2.4 MB/s | 103 kB 00:00 2024-03-25T16:25:33.493 INFO:teuthology.orchestra.run.smithi183.stdout:(13/36): python3-pycparser-2.20-6.el9.noarch.rp 3.6 MB/s | 124 kB 00:00 2024-03-25T16:25:33.535 INFO:teuthology.orchestra.run.smithi183.stdout:(14/36): python3-urllib3-1.26.5-5.el9.noarch.rp 4.4 MB/s | 187 kB 00:00 2024-03-25T16:25:34.294 INFO:teuthology.orchestra.run.smithi183.stdout:(15/36): python3-requests-2.25.1-8.el9.noarch.r 165 kB/s | 125 kB 00:00 2024-03-25T16:25:34.377 INFO:teuthology.orchestra.run.smithi183.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 269 kB/s | 22 kB 00:00 2024-03-25T16:25:34.603 INFO:teuthology.orchestra.run.smithi183.stdout:(17/36): luarocks-3.9.2-1.el9.noarch.rpm 672 kB/s | 151 kB 00:00 2024-03-25T16:25:34.796 INFO:teuthology.orchestra.run.smithi183.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 1.0 MB/s | 43 kB 00:00 2024-03-25T16:25:34.796 INFO:teuthology.orchestra.run.smithi183.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 2.5 MB/s | 172 kB 00:00 2024-03-25T16:25:34.803 INFO:teuthology.orchestra.run.smithi183.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.8 MB/s | 358 kB 00:00 2024-03-25T16:25:34.845 INFO:teuthology.orchestra.run.smithi183.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 256 kB/s | 11 kB 00:00 2024-03-25T16:25:34.887 INFO:teuthology.orchestra.run.smithi183.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 426 kB/s | 18 kB 00:00 2024-03-25T16:25:34.929 INFO:teuthology.orchestra.run.smithi183.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 556 kB/s | 23 kB 00:00 2024-03-25T16:25:34.980 INFO:teuthology.orchestra.run.smithi183.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 390 kB/s | 19 kB 00:00 2024-03-25T16:25:35.029 INFO:teuthology.orchestra.run.smithi183.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 395 kB/s | 20 kB 00:00 2024-03-25T16:25:35.088 INFO:teuthology.orchestra.run.smithi183.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 793 kB/s | 46 kB 00:00 2024-03-25T16:25:35.139 INFO:teuthology.orchestra.run.smithi183.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-25T16:25:35.197 INFO:teuthology.orchestra.run.smithi183.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 4.6 MB/s | 272 kB 00:00 2024-03-25T16:25:35.256 INFO:teuthology.orchestra.run.smithi183.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-25T16:25:35.315 INFO:teuthology.orchestra.run.smithi183.stdout:(30/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.5 MB/s | 90 kB 00:00 2024-03-25T16:25:35.330 INFO:teuthology.orchestra.run.smithi007.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 422 kB/s | 17 MB 00:40 2024-03-25T16:25:35.335 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:35.335 INFO:teuthology.orchestra.run.smithi007.stdout:Total 673 kB/s | 30 MB 00:45 2024-03-25T16:25:35.359 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:25:35.365 INFO:teuthology.orchestra.run.smithi183.stdout:(31/36): python3-tempora-5.0.0-2.el9.noarch.rpm 717 kB/s | 36 kB 00:00 2024-03-25T16:25:35.403 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:25:35.403 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:25:35.423 INFO:teuthology.orchestra.run.smithi183.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.9 MB/s | 230 kB 00:00 2024-03-25T16:25:35.490 INFO:teuthology.orchestra.run.smithi183.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 6.3 MB/s | 427 kB 00:00 2024-03-25T16:25:35.549 INFO:teuthology.orchestra.run.smithi183.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-25T16:25:35.815 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:25:35.815 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:25:37.174 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:25:37.240 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:25:37.278 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:25:37.309 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:25:37.385 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:25:37.434 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:25:37.466 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:25:37.563 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:25:37.648 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:25:37.694 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:25:37.770 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:25:37.821 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:25:37.858 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:25:37.926 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:25:37.989 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:25:38.185 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:25:38.238 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:25:38.276 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:25:38.311 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:25:38.341 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:25:38.421 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:25:38.459 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:25:38.493 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:25:38.534 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:25:38.577 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:25:38.620 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:25:38.715 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:25:38.751 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:25:38.807 INFO:teuthology.orchestra.run.smithi183.stdout:(35/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86 401 kB/s | 4.7 MB 00:11 2024-03-25T16:25:38.828 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:25:38.977 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:25:39.109 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:25:39.275 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:25:39.305 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:25:39.330 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:25:39.330 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:39.331 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:25:39.331 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:25:39.331 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:25:39.331 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:40.856 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:25:40.883 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:25:40.883 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:40.883 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:25:40.883 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:25:40.883 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:25:40.883 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:41.349 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:25:41.372 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:25:41.372 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:41.372 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:25:41.372 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:25:41.372 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:25:41.373 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:41.592 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:25:41.614 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:25:41.615 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:41.615 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:25:41.615 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:25:41.615 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:25:41.615 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:41.684 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:25:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:25:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:25:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:25:43.319 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:25:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:25:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:25:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:25:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:25:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:25:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:25:43.320 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:25:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:25:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:25:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:25:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:25:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:25:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:25:43.321 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:25:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:25:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:25:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:25:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:25:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:25:43.322 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:25:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:25:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:25:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:25:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:25:43.323 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:25:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:25:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:25:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:25:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:25:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:25:43.324 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:25:43.325 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:25:44.209 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:25:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:44.210 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:25:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:44.210 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:25:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:25:44.211 INFO:teuthology.orchestra.run.smithi007.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:25:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:25:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:25:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:25:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:25:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:25:44.212 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:25:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:25:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:25:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:25:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:25:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:25:44.213 INFO:teuthology.orchestra.run.smithi007.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:25:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:25:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:25:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:25:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:25:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:25:44.214 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:25:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:25:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:25:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:25:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:25:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:25:44.215 INFO:teuthology.orchestra.run.smithi007.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:25:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:25:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:25:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:25:44.216 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:44.216 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:25:44.566 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base 2024-03-25T16:25:45.140 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:25:45.240 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:45.295 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:25:45.296 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:25:45.296 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:25:45.362 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install cephadm 2024-03-25T16:25:45.931 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:02:59 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:25:46.085 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:25:46.086 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:25:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:25:46.086 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:25:46.086 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:25:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:25:46.086 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:46.086 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:25:46.087 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:25:46.087 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:25:46.087 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:46.087 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 747 k 2024-03-25T16:25:46.087 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 754 k 2024-03-25T16:25:46.087 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:25:48.003 INFO:teuthology.orchestra.run.smithi007.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 390 kB/s | 747 kB 00:01 2024-03-25T16:25:48.004 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:48.004 INFO:teuthology.orchestra.run.smithi007.stdout:Total 390 kB/s | 747 kB 00:01 2024-03-25T16:25:48.004 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:25:48.011 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:25:48.011 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:25:48.017 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:25:48.018 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:25:48.097 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:25:48.507 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:48.587 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:48.975 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:49.244 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:25:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:49.245 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:25:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:25:49.245 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:49.245 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:25:49.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:25:50.019 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:04 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:25:50.175 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:25:50.175 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 145 k 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 436 k 2024-03-25T16:25:50.176 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:25:50.951 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 188 kB/s | 145 kB 00:00 2024-03-25T16:25:50.952 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:25:50.952 INFO:teuthology.orchestra.run.smithi007.stdout:Total 187 kB/s | 145 kB 00:00 2024-03-25T16:25:50.952 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:25:50.961 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:25:50.962 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:25:51.020 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:25:51.020 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:25:51.163 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:25:51.235 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:51.260 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:51.260 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:25:51.260 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:25:51.260 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:52.202 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:25:52.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:52.202 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:25:52.202 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:25:52.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:52.202 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:25:52.423 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr 2024-03-25T16:25:53.001 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:07 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:25:53.102 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:25:53.157 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:25:53.159 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:25:53.159 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:25:53.226 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:25:53.796 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:07 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:25:53.951 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:25:53.952 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout:====================================================================================== 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-03-25T16:25:53.953 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:25:53.954 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.9 M 2024-03-25T16:25:53.954 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-03-25T16:25:53.954 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:01.371 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 2.1 kB/s | 15 kB 00:07 2024-03-25T16:26:01.396 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 4.3 kB/s | 31 kB 00:07 2024-03-25T16:26:01.530 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.7 MB/s | 238 kB 00:00 2024-03-25T16:26:01.589 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-25T16:26:01.822 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 236 kB/s | 1.8 MB 00:07 2024-03-25T16:26:02.123 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 65 kB/s | 19 kB 00:00 2024-03-25T16:26:02.206 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 888 kB/s | 548 kB 00:00 2024-03-25T16:26:02.315 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.3 MB/s | 144 kB 00:00 2024-03-25T16:26:02.373 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): protobuf-compiler-3.14.0-13.el9.x86_64. 1.0 MB/s | 863 kB 00:00 2024-03-25T16:26:02.424 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 283 kB/s | 31 kB 00:00 2024-03-25T16:26:02.782 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 3.1 MB/s | 2.0 MB 00:00 2024-03-25T16:26:02.815 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 427 kB/s | 188 kB 00:00 2024-03-25T16:26:02.819 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:02.819 INFO:teuthology.orchestra.run.smithi007.stdout:Total 792 kB/s | 6.9 MB 00:08 2024-03-25T16:26:02.840 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:02.860 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:02.860 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:03.040 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:03.041 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:03.698 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:03.784 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:26:03.819 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:26:03.867 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:26:03.904 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:26:04.005 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:26:04.135 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:26:04.319 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:26:04.368 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:26:04.396 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:26:04.427 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:26:04.682 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:26:04.719 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:26:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:26:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:26:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:26:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:26:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:26:05.503 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:26:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:26:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:26:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:26:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:26:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:26:05.504 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:26:05.867 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:26:05.867 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:26:05.868 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:05.869 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:06.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:26:06.710 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:20 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:06.867 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:06.868 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-25T16:26:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:26:06.868 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-25T16:26:06.868 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:26:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:26:06.868 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:26:06.868 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout:============================================================================================ 2024-03-25T16:26:06.869 INFO:teuthology.orchestra.run.smithi007.stdout:Install 12 Packages 2024-03-25T16:26:06.870 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:06.870 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 37 M 2024-03-25T16:26:06.870 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 209 M 2024-03-25T16:26:06.870 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:07.314 INFO:teuthology.orchestra.run.smithi007.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 305 kB/s | 32 kB 00:00 2024-03-25T16:26:07.364 INFO:teuthology.orchestra.run.smithi007.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 664 kB/s | 17 kB 00:00 2024-03-25T16:26:07.390 INFO:teuthology.orchestra.run.smithi007.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 1.3 MB/s | 34 kB 00:00 2024-03-25T16:26:07.557 INFO:teuthology.orchestra.run.smithi007.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 28 MB/s | 4.7 MB 00:00 2024-03-25T16:26:07.690 INFO:teuthology.orchestra.run.smithi007.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-25T16:26:07.749 INFO:teuthology.orchestra.run.smithi007.stdout:(6/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 5.5 MB/s | 3.0 MB 00:00 2024-03-25T16:26:07.774 INFO:teuthology.orchestra.run.smithi007.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-25T16:26:07.950 INFO:teuthology.orchestra.run.smithi007.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T16:26:08.000 INFO:teuthology.orchestra.run.smithi007.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.0 MB/s | 195 kB 00:00 2024-03-25T16:26:08.050 INFO:teuthology.orchestra.run.smithi007.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 5.1 MB/s | 245 kB 00:00 2024-03-25T16:26:08.201 INFO:teuthology.orchestra.run.smithi007.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-25T16:26:12.902 INFO:teuthology.orchestra.run.smithi183.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 403 kB/s | 17 MB 00:42 2024-03-25T16:26:12.907 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:12.907 INFO:teuthology.orchestra.run.smithi183.stdout:Total 646 kB/s | 30 MB 00:47 2024-03-25T16:26:12.932 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:26:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:26:12.975 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:26:13.418 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:26:13.418 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:26:14.733 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:26:14.788 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-25T16:26:14.838 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-25T16:26:14.865 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-25T16:26:14.936 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-25T16:26:14.981 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-25T16:26:15.013 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-25T16:26:15.105 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-25T16:26:15.192 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-25T16:26:15.241 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-25T16:26:15.339 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-25T16:26:15.392 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-25T16:26:15.430 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-25T16:26:15.479 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-25T16:26:15.535 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-25T16:26:15.728 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-25T16:26:15.776 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-25T16:26:15.806 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-25T16:26:15.841 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-25T16:26:15.871 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-25T16:26:15.948 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-25T16:26:15.990 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:26:16.023 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-25T16:26:16.064 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-25T16:26:16.099 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-25T16:26:16.133 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-25T16:26:16.223 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-25T16:26:16.264 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-25T16:26:16.336 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-25T16:26:16.475 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-25T16:26:16.600 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:26:16.764 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-25T16:26:16.794 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:26:16.819 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-25T16:26:16.819 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:16.820 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-25T16:26:16.820 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:26:16.820 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-25T16:26:16.820 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:18.262 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:26:18.288 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-25T16:26:18.288 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:18.288 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-25T16:26:18.288 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:26:18.288 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-25T16:26:18.288 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:18.729 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:26:18.753 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-25T16:26:18.753 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:18.753 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-25T16:26:18.753 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:26:18.753 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-25T16:26:18.753 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:18.955 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:26:18.978 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-25T16:26:18.978 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:18.978 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-25T16:26:18.978 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:26:18.978 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-25T16:26:18.978 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:19.044 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:26:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-25T16:26:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-25T16:26:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-25T16:26:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-25T16:26:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-25T16:26:20.722 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-25T16:26:20.723 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-25T16:26:20.724 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-25T16:26:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-25T16:26:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-25T16:26:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-25T16:26:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-25T16:26:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-25T16:26:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-25T16:26:20.725 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-25T16:26:20.726 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-25T16:26:21.775 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: lua-5.4.4-4.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-25T16:26:21.776 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-ply-3.11-14.el9.noarch 2024-03-25T16:26:21.777 INFO:teuthology.orchestra.run.smithi183.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:21.778 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:26:22.123 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base 2024-03-25T16:26:22.689 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:36 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:22.786 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:22.838 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:26:22.839 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:26:22.839 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:26:22.909 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install cephadm 2024-03-25T16:26:23.468 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:37 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:23.617 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:26:23.617 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:26:23.617 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 747 k 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 754 k 2024-03-25T16:26:23.618 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:26:25.838 INFO:teuthology.orchestra.run.smithi183.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 337 kB/s | 747 kB 00:02 2024-03-25T16:26:25.839 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:25.839 INFO:teuthology.orchestra.run.smithi183.stdout:Total 336 kB/s | 747 kB 00:02 2024-03-25T16:26:25.840 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:26:25.845 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:26:25.845 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:26:25.851 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:26:25.852 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:26:25.926 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:26:26.342 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:26.427 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:26.814 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:26.931 INFO:teuthology.orchestra.run.smithi007.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 384 kB/s | 7.4 MB 00:19 2024-03-25T16:26:26.934 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:26.935 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.8 MB/s | 37 MB 00:20 2024-03-25T16:26:26.982 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:27.005 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:27.006 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:27.095 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-25T16:26:27.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:27.095 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:26:27.095 INFO:teuthology.orchestra.run.smithi183.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:27.095 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:27.095 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:26:27.302 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache 2024-03-25T16:26:27.380 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:27.380 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:27.879 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:41 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:28.033 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:26:28.034 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:26:28.034 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T16:26:28.034 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:26:28.034 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:26:28.034 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 145 k 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 436 k 2024-03-25T16:26:28.035 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:26:28.107 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:28.180 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:26:28.219 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:26:28.326 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:28.356 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:26:28.540 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:28.572 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:28.655 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:26:28.769 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 198 kB/s | 145 kB 00:00 2024-03-25T16:26:28.770 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:28.770 INFO:teuthology.orchestra.run.smithi183.stdout:Total 198 kB/s | 145 kB 00:00 2024-03-25T16:26:28.770 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:26:28.778 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:26:28.778 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:26:28.830 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:26:28.830 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:26:28.970 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:26:29.037 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:26:29.061 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:26:29.061 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:29.061 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-25T16:26:29.062 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:29.339 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:26:29.455 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:26:30.034 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-25T16:26:30.034 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:30.034 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:26:30.034 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:30.034 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:30.034 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:26:30.289 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr 2024-03-25T16:26:30.770 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:26:30.863 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:44 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:30.963 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:31.014 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:26:31.015 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:26:31.015 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:26:31.092 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-dashboard 2024-03-25T16:26:31.655 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:45 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:31.804 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-25T16:26:31.805 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout:====================================================================================== 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-25T16:26:31.806 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:31.807 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.9 M 2024-03-25T16:26:31.807 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-25T16:26:31.807 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:26:31.847 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:26:31.880 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:26:32.672 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 34 kB/s | 15 kB 00:00 2024-03-25T16:26:32.698 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 66 kB/s | 31 kB 00:00 2024-03-25T16:26:32.848 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.5 MB/s | 238 kB 00:00 2024-03-25T16:26:32.882 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 4.8 MB/s | 1.0 MB 00:00 2024-03-25T16:26:33.073 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.8 MB/s | 863 kB 00:00 2024-03-25T16:26:33.190 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 166 kB/s | 19 kB 00:00 2024-03-25T16:26:33.216 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.6 MB/s | 548 kB 00:00 2024-03-25T16:26:33.358 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 12 MB/s | 2.0 MB 00:00 2024-03-25T16:26:33.392 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 822 kB/s | 144 kB 00:00 2024-03-25T16:26:33.417 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 521 kB/s | 31 kB 00:00 2024-03-25T16:26:33.484 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.0 MB/s | 188 kB 00:00 2024-03-25T16:26:34.374 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:26:34.374 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:26:34.374 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:26:34.374 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:34.374 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:26:34.374 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:34.374 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:34.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:26:34.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:26:34.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:26:34.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:26:34.375 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:26:34.803 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:26:34.803 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:34.803 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:34.803 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:34.803 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:34.804 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:35.142 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:26:35.722 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:49 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:35.880 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:35.881 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:26:35.881 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:26:35.882 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout:Install 13 Packages 2024-03-25T16:26:35.883 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:35.884 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 2.0 M 2024-03-25T16:26:35.884 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 26 M 2024-03-25T16:26:35.884 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:36.048 INFO:teuthology.orchestra.run.smithi007.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 204 kB/s | 24 kB 00:00 2024-03-25T16:26:36.090 INFO:teuthology.orchestra.run.smithi007.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 112 kB/s | 18 kB 00:00 2024-03-25T16:26:36.140 INFO:teuthology.orchestra.run.smithi007.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.5 MB/s | 174 kB 00:00 2024-03-25T16:26:36.166 INFO:teuthology.orchestra.run.smithi007.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-25T16:26:36.199 INFO:teuthology.orchestra.run.smithi007.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-25T16:26:36.216 INFO:teuthology.orchestra.run.smithi007.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 872 kB/s | 43 kB 00:00 2024-03-25T16:26:36.450 INFO:teuthology.orchestra.run.smithi007.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 129 kB/s | 32 kB 00:00 2024-03-25T16:26:36.467 INFO:teuthology.orchestra.run.smithi007.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-25T16:26:36.550 INFO:teuthology.orchestra.run.smithi007.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 84 kB/s | 52 kB 00:00 2024-03-25T16:26:36.693 INFO:teuthology.orchestra.run.smithi007.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 837 kB/s | 203 kB 00:00 2024-03-25T16:26:36.801 INFO:teuthology.orchestra.run.smithi007.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 828 kB/s | 90 kB 00:00 2024-03-25T16:26:36.859 INFO:teuthology.orchestra.run.smithi007.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 191 kB/s | 59 kB 00:00 2024-03-25T16:26:36.975 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 384 kB/s | 1.8 MB 00:04 2024-03-25T16:26:36.979 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:36.980 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.3 MB/s | 6.9 MB 00:05 2024-03-25T16:26:36.999 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:26:37.019 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:26:37.019 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:26:37.193 INFO:teuthology.orchestra.run.smithi007.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.4 MB/s | 1.0 MB 00:00 2024-03-25T16:26:37.195 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:37.195 INFO:teuthology.orchestra.run.smithi007.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-25T16:26:37.196 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:37.202 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:26:37.203 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:26:37.212 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:37.212 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:37.314 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:37.316 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:37.720 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:26:37.801 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-25T16:26:37.834 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:37.835 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-25T16:26:37.882 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-25T16:26:37.913 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-25T16:26:38.016 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-25T16:26:38.028 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:26:38.087 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:26:38.138 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-25T16:26:38.152 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:26:38.188 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:26:38.229 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:26:38.231 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-25T16:26:38.284 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-25T16:26:38.313 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:26:38.327 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:26:38.344 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-25T16:26:38.437 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:26:38.490 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:26:38.584 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-25T16:26:38.619 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:26:39.264 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:26:39.313 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:26:39.347 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:26:39.396 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:26:39.424 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:26:39.450 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-25T16:26:39.451 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-25T16:26:39.790 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-25T16:26:39.790 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:39.790 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:26:39.790 INFO:teuthology.orchestra.run.smithi183.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:39.791 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:26:40.065 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:26:40.228 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:26:40.580 INFO:teuthology.orchestra.run.smithi007.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:40.581 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:40.625 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:03:54 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:40.776 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-25T16:26:40.777 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout:============================================================================================ 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout:Install 12 Packages 2024-03-25T16:26:40.778 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:26:40.779 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 37 M 2024-03-25T16:26:40.779 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 209 M 2024-03-25T16:26:40.779 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:26:40.856 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:26:41.165 INFO:teuthology.orchestra.run.smithi183.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 257 kB/s | 32 kB 00:00 2024-03-25T16:26:41.223 INFO:teuthology.orchestra.run.smithi183.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 493 kB/s | 17 kB 00:00 2024-03-25T16:26:41.282 INFO:teuthology.orchestra.run.smithi183.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 584 kB/s | 34 kB 00:00 2024-03-25T16:26:41.431 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:03:55 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:41.466 INFO:teuthology.orchestra.run.smithi183.stdout:(4/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 26 MB/s | 4.7 MB 00:00 2024-03-25T16:26:41.587 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:26:41.588 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:26:41.589 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:26:41.589 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:26:41.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:41.589 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:26:41.589 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:26:41.589 INFO:teuthology.orchestra.run.smithi007.stdout:Install 6 Packages 2024-03-25T16:26:41.589 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:41.590 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 6.8 M 2024-03-25T16:26:41.590 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 33 M 2024-03-25T16:26:41.590 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:41.624 INFO:teuthology.orchestra.run.smithi183.stdout:(5/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 30 MB/s | 4.8 MB 00:00 2024-03-25T16:26:41.666 INFO:teuthology.orchestra.run.smithi183.stdout:(6/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 5.2 MB/s | 217 kB 00:00 2024-03-25T16:26:41.725 INFO:teuthology.orchestra.run.smithi183.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 4.4 MB/s | 3.0 MB 00:00 2024-03-25T16:26:41.900 INFO:teuthology.orchestra.run.smithi183.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.5 MB/s | 803 kB 00:00 2024-03-25T16:26:41.942 INFO:teuthology.orchestra.run.smithi183.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 4.7 MB/s | 195 kB 00:00 2024-03-25T16:26:42.276 INFO:teuthology.orchestra.run.smithi183.stdout:(10/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 26 MB/s | 16 MB 00:00 2024-03-25T16:26:42.431 INFO:teuthology.orchestra.run.smithi007.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 766 kB/s | 248 kB 00:00 2024-03-25T16:26:42.668 INFO:teuthology.orchestra.run.smithi183.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 339 kB/s | 245 kB 00:00 2024-03-25T16:26:42.931 INFO:teuthology.orchestra.run.smithi007.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 1.1 MB/s | 548 kB 00:00 2024-03-25T16:26:42.998 INFO:teuthology.orchestra.run.smithi007.stdout:(3/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 155 kB/s | 138 kB 00:00 2024-03-25T16:26:43.024 INFO:teuthology.orchestra.run.smithi007.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 633 kB/s | 58 kB 00:00 2024-03-25T16:26:43.107 INFO:teuthology.orchestra.run.smithi007.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 5.8 MB/s | 5.8 MB 00:00 2024-03-25T16:26:43.366 INFO:teuthology.orchestra.run.smithi007.stdout:(6/6): python3-typing-extensions-4.4.0-2.el9.no 139 kB/s | 51 kB 00:00 2024-03-25T16:26:43.370 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:43.370 INFO:teuthology.orchestra.run.smithi007.stdout:Total 3.8 MB/s | 6.8 MB 00:01 2024-03-25T16:26:43.385 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:43.397 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:43.398 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:43.492 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:43.493 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:43.800 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:43.964 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:26:44.019 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:26:44.572 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:26:44.656 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:26:44.734 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:26:44.783 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:26:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:26:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:26:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:26:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:26:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:26:45.586 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:26:45.885 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:26:45.885 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:45.886 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:46.128 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse 2024-03-25T16:26:46.708 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:00 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:46.865 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:46.865 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:46.866 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 964 k 2024-03-25T16:26:46.867 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.8 M 2024-03-25T16:26:46.867 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:47.001 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 586 kB/s | 79 kB 00:00 2024-03-25T16:26:48.984 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 418 kB/s | 885 kB 00:02 2024-03-25T16:26:48.985 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:48.986 INFO:teuthology.orchestra.run.smithi007.stdout:Total 455 kB/s | 964 kB 00:02 2024-03-25T16:26:48.986 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:49.011 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:49.011 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:49.090 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:49.090 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:49.269 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:49.406 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:26:49.478 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:26:50.190 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:26:50.190 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:26:50.513 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:26:50.513 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:50.513 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:50.513 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:26:50.513 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:50.514 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:50.797 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-volume 2024-03-25T16:26:51.383 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:05 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:51.540 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repository Size 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:26:51.541 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:51.542 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:26:51.542 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:51.542 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-25T16:26:51.542 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:51.542 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 332 k 2024-03-25T16:26:51.542 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.5 M 2024-03-25T16:26:51.542 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:51.789 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 278 kB/s | 69 kB 00:00 2024-03-25T16:26:52.648 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 239 kB/s | 264 kB 00:01 2024-03-25T16:26:52.649 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:52.649 INFO:teuthology.orchestra.run.smithi007.stdout:Total 300 kB/s | 332 kB 00:01 2024-03-25T16:26:52.650 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:52.660 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:52.661 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:52.707 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:52.707 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:52.883 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:53.066 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:26:53.100 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:26:53.115 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:26:53.115 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:26:53.115 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:26:53.115 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:53.858 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:26:54.202 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:26:54.202 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:54.202 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:54.203 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:26:54.203 INFO:teuthology.orchestra.run.smithi007.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:26:54.203 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:54.203 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:54.499 INFO:teuthology.orchestra.run.smithi183.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 562 kB/s | 7.4 MB 00:13 2024-03-25T16:26:54.501 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:54.502 INFO:teuthology.orchestra.run.smithi183.stdout:Total 2.7 MB/s | 37 MB 00:13 2024-03-25T16:26:54.545 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:26:54.570 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:26:54.570 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:26:54.572 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel 2024-03-25T16:26:54.925 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:26:54.925 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:26:55.156 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:09 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:55.311 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:26:55.312 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:55.313 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:26:55.313 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:55.313 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 127 k 2024-03-25T16:26:55.313 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 456 k 2024-03-25T16:26:55.313 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:55.631 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:26:55.705 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-25T16:26:55.745 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-25T16:26:55.836 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:26:55.873 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-25T16:26:55.878 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 224 kB/s | 127 kB 00:00 2024-03-25T16:26:55.879 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:55.879 INFO:teuthology.orchestra.run.smithi007.stdout:Total 223 kB/s | 127 kB 00:00 2024-03-25T16:26:55.879 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:55.886 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:55.886 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:55.944 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:55.944 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:56.043 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:26:56.072 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:26:56.092 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:56.174 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-25T16:26:56.197 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:56.641 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:56.809 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-25T16:26:56.925 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-25T16:26:56.991 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:56.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:56.991 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:56.991 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:56.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:56.991 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:57.230 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2 2024-03-25T16:26:57.809 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:11 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:57.909 INFO:teuthology.orchestra.run.smithi007.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:26:57.966 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:57.967 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:26:57.967 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:26:58.036 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs-devel 2024-03-25T16:26:58.147 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-25T16:26:58.615 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:12 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:26:58.770 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:26:58.771 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:26:58.772 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:58.772 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 31 k 2024-03-25T16:26:58.772 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 140 k 2024-03-25T16:26:58.772 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:26:59.074 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 103 kB/s | 31 kB 00:00 2024-03-25T16:26:59.075 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:26:59.075 INFO:teuthology.orchestra.run.smithi007.stdout:Total 102 kB/s | 31 kB 00:00 2024-03-25T16:26:59.075 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:26:59.079 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:26:59.079 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:26:59.105 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:26:59.106 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:26:59.168 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-25T16:26:59.197 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:26:59.225 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:26:59.358 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:59.722 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:26:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:59.991 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:26:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:26:59.991 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:26:59.991 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:00.147 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2 2024-03-25T16:27:00.723 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:14 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:00.823 INFO:teuthology.orchestra.run.smithi007.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:00.878 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:27:00.879 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:00.933 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1 2024-03-25T16:27:01.511 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:15 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:01.611 INFO:teuthology.orchestra.run.smithi007.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:01.666 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:01.667 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:27:01.667 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:01.715 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-25T16:27:01.848 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-25T16:27:01.849 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-25T16:27:02.264 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-25T16:27:02.264 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:02.264 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:02.264 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:02.264 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:02.265 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:02.294 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:16 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:02.395 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:02.451 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:02.452 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:27:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:02.503 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw 2024-03-25T16:27:02.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-rook 2024-03-25T16:27:03.090 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:03.191 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:03.197 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:03.246 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:27:03.247 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:03.299 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs 2024-03-25T16:27:03.347 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:03.348 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T16:27:03.348 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:27:03.348 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-25T16:27:03.349 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-25T16:27:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-25T16:27:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:03.350 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:03.350 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T16:27:03.350 INFO:teuthology.orchestra.run.smithi183.stdout:Install 13 Packages 2024-03-25T16:27:03.350 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:03.351 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 2.0 M 2024-03-25T16:27:03.351 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 26 M 2024-03-25T16:27:03.351 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:03.556 INFO:teuthology.orchestra.run.smithi183.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 170 kB/s | 18 kB 00:00 2024-03-25T16:27:03.623 INFO:teuthology.orchestra.run.smithi183.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 141 kB/s | 24 kB 00:00 2024-03-25T16:27:03.682 INFO:teuthology.orchestra.run.smithi183.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.7 MB/s | 174 kB 00:00 2024-03-25T16:27:03.707 INFO:teuthology.orchestra.run.smithi183.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-25T16:27:03.732 INFO:teuthology.orchestra.run.smithi183.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.2 MB/s | 211 kB 00:00 2024-03-25T16:27:03.758 INFO:teuthology.orchestra.run.smithi183.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 854 kB/s | 43 kB 00:00 2024-03-25T16:27:03.877 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:17 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:03.976 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:03.991 INFO:teuthology.orchestra.run.smithi183.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 124 kB/s | 32 kB 00:00 2024-03-25T16:27:04.008 INFO:teuthology.orchestra.run.smithi183.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 60 kB/s | 15 kB 00:00 2024-03-25T16:27:04.025 INFO:teuthology.orchestra.run.smithi183.stdout:(9/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 91 kB/s | 52 kB 00:00 2024-03-25T16:27:04.030 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:04.032 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:27:04.032 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:04.089 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd 2024-03-25T16:27:04.234 INFO:teuthology.orchestra.run.smithi183.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 837 kB/s | 203 kB 00:00 2024-03-25T16:27:04.334 INFO:teuthology.orchestra.run.smithi183.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 897 kB/s | 90 kB 00:00 2024-03-25T16:27:04.351 INFO:teuthology.orchestra.run.smithi183.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 182 kB/s | 59 kB 00:00 2024-03-25T16:27:04.518 INFO:teuthology.orchestra.run.smithi183.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 2.0 MB/s | 1.0 MB 00:00 2024-03-25T16:27:04.520 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:04.520 INFO:teuthology.orchestra.run.smithi183.stdout:Total 1.8 MB/s | 2.0 MB 00:01 2024-03-25T16:27:04.520 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:04.534 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:04.534 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:04.626 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:04.626 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:04.666 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:18 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:04.765 INFO:teuthology.orchestra.run.smithi007.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:04.821 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:04.822 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T16:27:04.822 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:04.878 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse 2024-03-25T16:27:05.147 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:05.315 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-25T16:27:05.377 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-25T16:27:05.432 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-25T16:27:05.455 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:19 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:05.461 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-25T16:27:05.503 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-25T16:27:05.606 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-25T16:27:05.609 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:05.610 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:05.610 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:05.610 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:05.610 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:27:05.610 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 85 k 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 231 k 2024-03-25T16:27:05.611 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:27:05.706 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-25T16:27:05.754 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-25T16:27:06.492 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-25T16:27:06.535 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-25T16:27:06.568 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-25T16:27:06.569 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 89 kB/s | 85 kB 00:00 2024-03-25T16:27:06.570 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:06.570 INFO:teuthology.orchestra.run.smithi007.stdout:Total 89 kB/s | 85 kB 00:00 2024-03-25T16:27:06.570 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:27:06.579 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:27:06.579 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:27:06.618 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-25T16:27:06.640 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:27:06.640 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:27:06.641 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:27:06.785 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:27:06.912 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:07.269 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-25T16:27:07.463 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-25T16:27:07.547 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:07.547 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:07.547 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:27:07.547 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:07.547 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:07.548 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:07.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror 2024-03-25T16:27:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-25T16:27:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:07.821 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-25T16:27:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-25T16:27:07.821 INFO:teuthology.orchestra.run.smithi183.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:07.822 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:08.090 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-cephadm 2024-03-25T16:27:08.327 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:08.486 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:27:08.487 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:08.488 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:27:08.488 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:08.488 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 3.1 M 2024-03-25T16:27:08.488 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 13 M 2024-03-25T16:27:08.488 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:27:08.652 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:22 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:08.801 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-25T16:27:08.802 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-25T16:27:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-25T16:27:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-25T16:27:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:08.803 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:08.803 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T16:27:08.803 INFO:teuthology.orchestra.run.smithi183.stdout:Install 6 Packages 2024-03-25T16:27:08.803 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:08.804 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 6.8 M 2024-03-25T16:27:08.804 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 33 M 2024-03-25T16:27:08.804 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:09.288 INFO:teuthology.orchestra.run.smithi183.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.5 MB/s | 248 kB 00:00 2024-03-25T16:27:09.446 INFO:teuthology.orchestra.run.smithi183.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 18 MB/s | 5.8 MB 00:00 2024-03-25T16:27:09.472 INFO:teuthology.orchestra.run.smithi183.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.9 MB/s | 548 kB 00:00 2024-03-25T16:27:09.497 INFO:teuthology.orchestra.run.smithi183.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-03-25T16:27:09.522 INFO:teuthology.orchestra.run.smithi183.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 768 kB/s | 58 kB 00:00 2024-03-25T16:27:10.197 INFO:teuthology.orchestra.run.smithi183.stdout:(6/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 129 kB/s | 138 kB 00:01 2024-03-25T16:27:10.199 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:10.200 INFO:teuthology.orchestra.run.smithi183.stdout:Total 4.9 MB/s | 6.8 MB 00:01 2024-03-25T16:27:10.213 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:10.225 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:10.225 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:10.315 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:10.316 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:10.620 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:10.785 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-25T16:27:10.832 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-25T16:27:11.342 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-25T16:27:11.429 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-25T16:27:11.487 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-25T16:27:11.503 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:27:12.299 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-25T16:27:12.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-25T16:27:12.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-25T16:27:12.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-25T16:27:12.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-25T16:27:12.299 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-25T16:27:12.601 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:12.602 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:12.854 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse 2024-03-25T16:27:13.424 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:27 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:13.573 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:13.573 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 964 k 2024-03-25T16:27:13.574 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.8 M 2024-03-25T16:27:13.575 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:13.683 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 733 kB/s | 79 kB 00:00 2024-03-25T16:27:15.881 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 428 kB/s | 3.1 MB 00:07 2024-03-25T16:27:15.881 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:15.882 INFO:teuthology.orchestra.run.smithi007.stdout:Total 428 kB/s | 3.1 MB 00:07 2024-03-25T16:27:15.882 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:27:15.894 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:27:15.894 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:27:15.973 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:27:15.973 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:27:16.426 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:27:16.432 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 310 kB/s | 885 kB 00:02 2024-03-25T16:27:16.433 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:16.434 INFO:teuthology.orchestra.run.smithi183.stdout:Total 337 kB/s | 964 kB 00:02 2024-03-25T16:27:16.434 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:16.457 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:16.457 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:16.490 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:16.514 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:16.514 INFO:teuthology.orchestra.run.smithi007.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:16.514 INFO:teuthology.orchestra.run.smithi007.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:27:16.515 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:16.515 INFO:teuthology.orchestra.run.smithi007.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:16.515 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:16.533 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:16.533 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:16.723 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:16.858 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-25T16:27:16.925 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:17.635 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:27:17.636 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:27:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:17.762 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:17.763 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:27:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:17.763 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:17.763 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:17.905 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-25T16:27:17.905 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:17.905 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:17.905 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-25T16:27:17.905 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:17.905 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:18.018 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd 2024-03-25T16:27:18.184 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-volume 2024-03-25T16:27:18.602 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:32 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:18.757 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi007.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:18.758 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:27:18.759 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:18.759 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:27:18.759 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:18.759 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 171 k 2024-03-25T16:27:18.759 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 494 k 2024-03-25T16:27:18.759 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:27:18.766 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:32 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:18.918 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:18.919 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:18.919 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repository Size 2024-03-25T16:27:18.919 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:18.919 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:18.919 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-25T16:27:18.919 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:27:18.920 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-25T16:27:18.920 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:18.920 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:18.920 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:18.920 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-25T16:27:18.920 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:18.920 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 332 k 2024-03-25T16:27:18.921 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.5 M 2024-03-25T16:27:18.921 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:19.046 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 555 kB/s | 69 kB 00:00 2024-03-25T16:27:19.726 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 177 kB/s | 171 kB 00:00 2024-03-25T16:27:19.726 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:19.727 INFO:teuthology.orchestra.run.smithi007.stdout:Total 177 kB/s | 171 kB 00:00 2024-03-25T16:27:19.727 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:27:19.737 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:27:19.737 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:27:19.799 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:27:19.800 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:27:19.957 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:27:20.070 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:20.510 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:20.778 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:20.778 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:20.779 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:27:20.779 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:20.779 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:20.779 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:27:20.928 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 131 kB/s | 264 kB 00:02 2024-03-25T16:27:20.929 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:20.929 INFO:teuthology.orchestra.run.smithi183.stdout:Total 165 kB/s | 332 kB 00:02 2024-03-25T16:27:20.930 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:20.939 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:20.940 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:20.951 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:27:20.986 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:20.986 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:21.133 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:21.309 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-25T16:27:21.339 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:27:21.355 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-25T16:27:21.355 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:21.355 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-25T16:27:21.355 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:21.523 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:04:35 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:21.680 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:27:21.680 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:21.680 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:27:21.680 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout:Installing dependencies: 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout:Install 2 Packages 2024-03-25T16:27:21.681 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:27:21.682 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 91 M 2024-03-25T16:27:21.682 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 498 M 2024-03-25T16:27:21.682 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:27:22.093 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-25T16:27:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-25T16:27:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:22.425 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-25T16:27:22.425 INFO:teuthology.orchestra.run.smithi183.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-25T16:27:22.426 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:22.426 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:22.753 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel 2024-03-25T16:27:23.332 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:37 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:23.481 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:23.482 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:23.482 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T16:27:23.482 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:23.482 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:23.482 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 127 k 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 456 k 2024-03-25T16:27:23.483 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:24.159 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 187 kB/s | 127 kB 00:00 2024-03-25T16:27:24.160 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:24.160 INFO:teuthology.orchestra.run.smithi183.stdout:Total 187 kB/s | 127 kB 00:00 2024-03-25T16:27:24.160 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:24.165 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:24.165 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:24.222 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:24.222 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:24.367 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:24.473 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:24.966 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:25.249 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:25.249 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:25.249 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:25.249 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:25.249 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:25.249 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:25.486 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2 2024-03-25T16:27:26.060 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:26.158 INFO:teuthology.orchestra.run.smithi183.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:26.214 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:26.215 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:27:26.215 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:26.282 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs-devel 2024-03-25T16:27:26.847 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:40 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:26.996 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 31 k 2024-03-25T16:27:26.997 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 140 k 2024-03-25T16:27:26.998 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:28.004 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 31 kB/s | 31 kB 00:01 2024-03-25T16:27:28.004 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:28.005 INFO:teuthology.orchestra.run.smithi183.stdout:Total 31 kB/s | 31 kB 00:01 2024-03-25T16:27:28.005 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:28.009 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:28.009 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:28.039 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:28.039 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:28.175 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:28.277 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:28.645 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:28.906 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:28.906 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:28.906 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:28.906 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:28.906 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:28.906 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:29.059 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2 2024-03-25T16:27:29.644 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:43 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:29.748 INFO:teuthology.orchestra.run.smithi183.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:29.803 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:29.804 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:27:29.804 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:29.861 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1 2024-03-25T16:27:30.462 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:44 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:30.563 INFO:teuthology.orchestra.run.smithi183.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:30.617 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:30.617 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:27:30.618 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:30.666 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados 2024-03-25T16:27:31.226 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:31.323 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:31.374 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:31.375 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:27:31.375 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:31.421 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw 2024-03-25T16:27:31.990 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:45 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:32.087 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:32.138 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:32.139 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:27:32.139 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:32.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs 2024-03-25T16:27:32.765 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:46 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:32.861 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:32.913 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:32.914 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:27:32.914 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:32.963 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd 2024-03-25T16:27:33.527 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:47 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:33.624 INFO:teuthology.orchestra.run.smithi183.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T16:27:33.675 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:33.676 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T16:27:33.677 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:33.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse 2024-03-25T16:27:34.302 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:48 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:34.452 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:34.453 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:34.453 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:34.453 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:34.453 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 85 k 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 231 k 2024-03-25T16:27:34.454 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:35.516 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 80 kB/s | 85 kB 00:01 2024-03-25T16:27:35.516 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:35.516 INFO:teuthology.orchestra.run.smithi183.stdout:Total 80 kB/s | 85 kB 00:01 2024-03-25T16:27:35.517 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:35.525 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:35.525 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:35.584 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:35.585 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:35.726 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:35.830 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:36.274 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:36.581 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:36.581 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:36.581 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:36.582 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:36.582 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:36.582 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:36.799 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror 2024-03-25T16:27:37.368 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:04:51 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:37.523 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:37.523 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 3.1 M 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 13 M 2024-03-25T16:27:37.524 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:45.254 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 409 kB/s | 3.1 MB 00:07 2024-03-25T16:27:45.254 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:45.254 INFO:teuthology.orchestra.run.smithi183.stdout:Total 409 kB/s | 3.1 MB 00:07 2024-03-25T16:27:45.254 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:45.263 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:45.263 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:45.339 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:45.339 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:45.757 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:45.835 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:45.859 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:45.859 INFO:teuthology.orchestra.run.smithi183.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-25T16:27:45.859 INFO:teuthology.orchestra.run.smithi183.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-25T16:27:45.859 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:45.859 INFO:teuthology.orchestra.run.smithi183.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-25T16:27:45.860 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:46.955 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:46.955 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:46.955 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:47.222 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd 2024-03-25T16:27:47.801 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:05:01 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:47.951 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout: Package Architecture Version Repository Size 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:47.952 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:27:47.953 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:47.953 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 171 k 2024-03-25T16:27:47.953 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 494 k 2024-03-25T16:27:47.953 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:27:48.846 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 192 kB/s | 171 kB 00:00 2024-03-25T16:27:48.847 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:27:48.847 INFO:teuthology.orchestra.run.smithi183.stdout:Total 192 kB/s | 171 kB 00:00 2024-03-25T16:27:48.847 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:27:48.856 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:27:48.856 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:27:48.916 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:27:48.916 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:27:49.065 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:27:49.169 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:49.606 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:49.879 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T16:27:49.879 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:49.879 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:27:49.879 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:27:49.879 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:49.879 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:27:50.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base-debuginfo 2024-03-25T16:27:50.682 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:05:04 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:27:50.832 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:27:50.832 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T16:27:50.832 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:50.832 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:27:50.832 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 68 M 2024-03-25T16:27:50.833 INFO:teuthology.orchestra.run.smithi183.stdout:Installing dependencies: 2024-03-25T16:27:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 22 M 2024-03-25T16:27:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:50.833 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:27:50.833 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:27:50.833 INFO:teuthology.orchestra.run.smithi183.stdout:Install 2 Packages 2024-03-25T16:27:50.833 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:27:50.834 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 91 M 2024-03-25T16:27:50.834 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 498 M 2024-03-25T16:27:50.834 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:28:21.438 INFO:teuthology.orchestra.run.smithi007.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 385 kB/s | 22 MB 00:59 2024-03-25T16:28:52.840 INFO:teuthology.orchestra.run.smithi183.stdout:(1/2): ceph-debuginfo-19.0.0-1439.g5dcb190d.el9 371 kB/s | 22 MB 01:02 2024-03-25T16:28:56.266 INFO:teuthology.orchestra.run.smithi007.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 738 kB/s | 68 MB 01:34 2024-03-25T16:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout:Total 981 kB/s | 91 MB 01:34 2024-03-25T16:28:56.268 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:28:56.272 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:28:56.273 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:28:56.879 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:28:56.879 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:29:00.145 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:29:07.660 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:29:07.736 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:29:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:29:11.106 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:29:11.378 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:29:11.378 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:29:11.378 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:29:11.378 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:11.378 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:29:11.378 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:29:11.378 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:29:11.583 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:29:12.166 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:06:26 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:29:12.325 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:29:12.326 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:29:12.327 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 416 M 2024-03-25T16:29:12.327 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.7 G 2024-03-25T16:29:12.327 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:31:39.899 INFO:teuthology.orchestra.run.smithi183.stdout:(2/2): ceph-base-debuginfo-19.0.0-1439.g5dcb190 305 kB/s | 68 MB 03:49 2024-03-25T16:31:39.899 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:31:39.899 INFO:teuthology.orchestra.run.smithi183.stdout:Total 405 kB/s | 91 MB 03:49 2024-03-25T16:31:39.899 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T16:31:39.903 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T16:31:39.903 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T16:31:40.488 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T16:31:40.488 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T16:31:43.576 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T16:31:50.746 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-25T16:31:50.946 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:31:54.134 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 2/2 2024-03-25T16:31:54.135 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/2 2024-03-25T16:31:54.392 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-25T16:31:54.393 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:31:54.393 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T16:31:54.393 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:54.393 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:31:54.393 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:31:54.393 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T16:31:54.612 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-common-debuginfo 2024-03-25T16:31:55.173 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:09:09 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:31:55.320 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 416 M 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 416 M 2024-03-25T16:31:55.321 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.7 G 2024-03-25T16:31:55.322 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T16:47:56.192 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 379 kB/s | 416 MB 18:43 2024-03-25T16:47:56.251 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:47:56.251 INFO:teuthology.orchestra.run.smithi007.stdout:Total 379 kB/s | 416 MB 18:43 2024-03-25T16:47:56.252 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:47:56.252 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:47:56.252 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:47:58.783 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:47:58.783 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:48:43.369 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:48:43.470 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:48:51.694 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:48:51.977 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T16:48:51.977 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:48:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:48:51.977 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:48:51.977 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:48:51.977 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:48:52.217 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T16:48:52.795 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:26:06 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:48:52.948 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:48:52.949 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:48:52.949 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:48:52.949 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:48:52.949 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:48:52.949 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T16:48:52.949 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:48:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:48:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:======================================================================================== 2024-03-25T16:48:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:48:52.950 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:48:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.4 M 2024-03-25T16:48:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 6.4 M 2024-03-25T16:48:52.950 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:48:58.038 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 287 kB/s | 1.4 MB 00:05 2024-03-25T16:48:58.040 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:48:58.040 INFO:teuthology.orchestra.run.smithi007.stdout:Total 287 kB/s | 1.4 MB 00:05 2024-03-25T16:48:58.040 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:48:58.044 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:48:58.044 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:48:58.091 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:48:58.091 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:48:58.368 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:48:58.473 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:48:58.907 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:48:59.187 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T16:48:59.187 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:48:59.187 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:48:59.187 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:48:59.187 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:48:59.188 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:48:59.364 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T16:48:59.936 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:26:13 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:49:00.091 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:49:00.092 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:49:00.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:49:00.093 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 176 M 2024-03-25T16:49:00.093 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 753 M 2024-03-25T16:49:00.093 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T16:58:26.185 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 318 kB/s | 176 MB 09:26 2024-03-25T16:58:26.186 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T16:58:26.186 INFO:teuthology.orchestra.run.smithi007.stdout:Total 318 kB/s | 176 MB 09:26 2024-03-25T16:58:26.186 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T16:58:26.189 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T16:58:26.189 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T16:58:27.301 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T16:58:27.301 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T16:58:46.393 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T16:58:46.500 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:51.388 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T16:58:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:58:51.647 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T16:58:51.647 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T16:58:51.648 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:58:51.648 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T16:58:51.839 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-test-debuginfo 2024-03-25T16:58:52.412 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 0:36:06 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T16:58:52.567 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T16:58:52.568 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T16:58:52.569 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T16:58:52.569 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T16:58:52.569 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 672 M 2024-03-25T16:58:52.569 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 3.1 G 2024-03-25T16:58:52.569 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:01:42.411 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-common-debuginfo-19.0.0-1439.g5dcb190d.el9 238 kB/s | 416 MB 29:47 2024-03-25T17:01:42.412 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:01:42.412 INFO:teuthology.orchestra.run.smithi183.stdout:Total 238 kB/s | 416 MB 29:47 2024-03-25T17:01:42.412 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T17:01:42.415 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T17:01:42.415 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T17:01:44.894 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T17:01:44.894 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T17:02:27.104 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T17:02:27.211 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:35.104 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:35.355 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T17:02:35.355 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:35.356 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T17:02:35.356 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-common-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:02:35.356 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:35.356 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:02:35.609 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-immutable-object-cache-debuginfo 2024-03-25T17:02:36.175 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:39:50 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:02:36.323 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:02:36.323 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T17:02:36.323 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.4 M 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:======================================================================================== 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 1.4 M 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 6.4 M 2024-03-25T17:02:36.324 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T17:02:42.673 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-immutable-object-cache-debuginfo-19.0.0-14 230 kB/s | 1.4 MB 00:06 2024-03-25T17:02:42.674 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:02:42.674 INFO:teuthology.orchestra.run.smithi183.stdout:Total 230 kB/s | 1.4 MB 00:06 2024-03-25T17:02:42.674 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T17:02:42.678 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T17:02:42.678 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T17:02:42.722 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T17:02:42.722 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T17:02:42.980 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T17:02:43.087 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:43.497 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:43.768 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-immutable-object-cache-debuginfo-2:19.0.0-1439. 1/1 2024-03-25T17:02:43.768 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:43.768 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T17:02:43.768 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-immutable-object-cache-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:02:43.768 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:43.768 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:02:43.965 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-radosgw-debuginfo 2024-03-25T17:02:44.527 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:39:58 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 176 M 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T17:02:44.675 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:02:44.676 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T17:02:44.676 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:02:44.676 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 176 M 2024-03-25T17:02:44.676 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 753 M 2024-03-25T17:02:44.676 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T17:09:44.002 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-radosgw-debuginfo-19.0.0-1439.g5dcb190d.el 429 kB/s | 176 MB 06:59 2024-03-25T17:09:44.002 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:09:44.003 INFO:teuthology.orchestra.run.smithi183.stdout:Total 429 kB/s | 176 MB 06:59 2024-03-25T17:09:44.003 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T17:09:44.006 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T17:09:44.006 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T17:09:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T17:09:45.065 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T17:10:03.053 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T17:10:03.156 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:10:07.630 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:10:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x 1/1 2024-03-25T17:10:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:10:07.884 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T17:10:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-radosgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:10:07.884 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:10:07.884 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:10:08.099 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-test-debuginfo 2024-03-25T17:10:08.658 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 0:47:22 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:10:08.806 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 672 M 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:10:08.807 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 672 M 2024-03-25T17:10:08.808 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 3.1 G 2024-03-25T17:10:08.808 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T17:36:01.258 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 309 kB/s | 672 MB 37:08 2024-03-25T17:36:01.282 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:36:01.282 INFO:teuthology.orchestra.run.smithi007.stdout:Total 309 kB/s | 672 MB 37:08 2024-03-25T17:36:01.282 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:36:01.282 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:36:01.282 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:36:05.435 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:36:05.435 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:37:20.449 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:37:20.560 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:28.398 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:28.672 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:37:28.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:37:28.672 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:37:28.672 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:37:28.672 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:37:28.672 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:37:28.949 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:37:29.525 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:14:43 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:37:29.624 INFO:teuthology.orchestra.run.smithi007.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:37:29.679 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:37:29.681 INFO:teuthology.orchestra.run.smithi007.stdout:Nothing to do. 2024-03-25T17:37:29.681 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:37:29.748 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:37:30.323 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:14:44 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:37:30.478 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:37:30.479 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:37:30.480 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 20 M 2024-03-25T17:37:30.480 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 103 M 2024-03-25T17:37:30.480 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:38:26.429 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 361 kB/s | 20 MB 00:55 2024-03-25T17:38:26.430 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:38:26.430 INFO:teuthology.orchestra.run.smithi007.stdout:Total 361 kB/s | 20 MB 00:55 2024-03-25T17:38:26.430 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:38:26.434 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:38:26.434 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:38:26.590 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:38:26.590 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:38:28.952 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:38:29.062 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:30.027 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:38:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:38:30.293 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:38:30.293 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:38:30.294 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:38:30.294 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:38:30.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:38:31.014 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:15:45 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:38:31.168 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:38:31.169 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:38:31.170 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 48 M 2024-03-25T17:38:31.170 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 173 M 2024-03-25T17:38:31.170 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:40:51.453 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 348 kB/s | 48 MB 02:20 2024-03-25T17:40:51.453 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:40:51.454 INFO:teuthology.orchestra.run.smithi007.stdout:Total 348 kB/s | 48 MB 02:20 2024-03-25T17:40:51.454 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:40:51.457 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:40:51.457 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:40:51.783 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:40:51.784 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:40:56.815 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:40:56.911 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:58.276 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:40:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:40:58.535 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:40:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:40:58.535 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:40:58.535 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:40:58.697 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:40:59.272 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:18:13 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:40:59.426 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:40:59.427 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:40:59.427 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:40:59.427 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:40:59.427 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:40:59.427 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:40:59.427 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:40:59.428 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:40:59.428 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:40:59.428 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:40:59.428 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:40:59.428 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 99 M 2024-03-25T17:40:59.428 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 449 M 2024-03-25T17:40:59.428 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:41:26.220 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-test-debuginfo-19.0.0-1439.g5dcb190d.el9.x 367 kB/s | 672 MB 31:17 2024-03-25T17:41:26.221 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:41:26.221 INFO:teuthology.orchestra.run.smithi183.stdout:Total 367 kB/s | 672 MB 31:17 2024-03-25T17:41:26.221 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T17:41:26.226 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T17:41:26.226 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T17:41:30.195 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T17:41:30.195 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T17:42:40.997 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T17:42:41.110 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:42:50.371 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:42:50.622 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:42:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:42:50.623 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T17:42:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-test-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:42:50.623 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:42:50.623 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:42:50.907 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-base-debuginfo 2024-03-25T17:42:51.475 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:20:05 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:42:51.571 INFO:teuthology.orchestra.run.smithi183.stdout:Package ceph-base-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-25T17:42:51.624 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:42:51.625 INFO:teuthology.orchestra.run.smithi183.stdout:Nothing to do. 2024-03-25T17:42:51.625 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:42:51.699 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mgr-debuginfo 2024-03-25T17:42:52.263 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:20:06 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:42:52.411 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:42:52.411 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:42:52.411 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T17:42:52.412 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:42:52.413 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 20 M 2024-03-25T17:42:52.413 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 103 M 2024-03-25T17:42:52.413 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T17:43:36.274 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-mgr-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 460 kB/s | 20 MB 00:43 2024-03-25T17:43:36.275 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:43:36.275 INFO:teuthology.orchestra.run.smithi183.stdout:Total 460 kB/s | 20 MB 00:43 2024-03-25T17:43:36.275 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T17:43:36.279 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T17:43:36.279 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T17:43:36.427 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T17:43:36.427 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T17:43:38.649 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T17:43:38.822 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:39.743 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:43:39.996 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:43:39.997 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T17:43:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mgr-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:43:39.997 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:43:39.997 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:43:40.161 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mds-debuginfo 2024-03-25T17:43:40.722 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:20:54 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:43:40.872 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 48 M 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T17:43:40.873 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:43:40.874 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 48 M 2024-03-25T17:43:40.874 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 173 M 2024-03-25T17:43:40.874 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T17:44:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 551 kB/s | 99 MB 03:03 2024-03-25T17:44:02.453 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:Total 551 kB/s | 99 MB 03:03 2024-03-25T17:44:02.454 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:44:02.457 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:44:02.457 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:44:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:44:03.098 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:44:14.012 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:44:14.103 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:17.041 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:17.302 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:44:17.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:44:17.303 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:44:17.303 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:44:17.303 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:44:17.303 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:44:17.467 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:44:18.043 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:21:32 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:44:18.198 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:44:18.199 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:44:18.200 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 287 M 2024-03-25T17:44:18.200 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.2 G 2024-03-25T17:44:18.200 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:44:58.223 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-mds-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 632 kB/s | 48 MB 01:17 2024-03-25T17:44:58.224 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:44:58.224 INFO:teuthology.orchestra.run.smithi183.stdout:Total 632 kB/s | 48 MB 01:17 2024-03-25T17:44:58.224 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T17:44:58.227 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T17:44:58.227 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T17:44:58.539 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T17:44:58.539 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T17:45:03.264 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T17:45:03.361 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:04.664 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:04.914 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:45:04.914 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:45:04.914 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T17:45:04.915 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mds-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:45:04.915 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:45:04.915 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:45:05.084 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-mon-debuginfo 2024-03-25T17:45:05.649 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:22:19 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:45:05.798 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:45:05.799 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:45:05.799 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T17:45:05.799 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:45:05.799 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T17:45:05.800 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 M 2024-03-25T17:45:05.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:45:05.800 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T17:45:05.800 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:45:05.800 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T17:45:05.800 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:45:05.800 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 99 M 2024-03-25T17:45:05.801 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 449 M 2024-03-25T17:45:05.801 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T17:49:01.806 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-mon-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 427 kB/s | 99 MB 03:56 2024-03-25T17:49:01.807 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:49:01.807 INFO:teuthology.orchestra.run.smithi183.stdout:Total 427 kB/s | 99 MB 03:56 2024-03-25T17:49:01.807 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T17:49:01.810 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T17:49:01.810 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T17:49:02.417 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T17:49:02.418 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T17:49:12.687 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T17:49:12.786 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:15.586 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:15.844 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:49:15.845 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:49:15.845 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T17:49:15.845 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-mon-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:49:15.845 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:49:15.845 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T17:49:16.036 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-osd-debuginfo 2024-03-25T17:49:16.605 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:26:30 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:49:16.754 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 287 M 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T17:49:16.755 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T17:49:16.756 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 287 M 2024-03-25T17:49:16.756 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.2 G 2024-03-25T17:49:16.756 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T17:58:14.220 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 351 kB/s | 287 MB 13:56 2024-03-25T17:58:14.220 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:58:14.220 INFO:teuthology.orchestra.run.smithi007.stdout:Total 351 kB/s | 287 MB 13:56 2024-03-25T17:58:14.220 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:58:14.223 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:58:14.223 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:58:16.013 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:58:16.013 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:58:47.124 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:58:47.265 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:54.897 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:55.178 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T17:58:55.178 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:58:55.178 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:58:55.178 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:58:55.178 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:58:55.178 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:58:55.379 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T17:58:55.958 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:36:09 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:58:56.113 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.9 M 2024-03-25T17:58:56.114 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 28 M 2024-03-25T17:58:56.115 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:59:02.282 INFO:teuthology.orchestra.run.smithi007.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 987 kB/s | 5.9 MB 00:06 2024-03-25T17:59:02.283 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:02.283 INFO:teuthology.orchestra.run.smithi007.stdout:Total 986 kB/s | 5.9 MB 00:06 2024-03-25T17:59:02.284 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:59:02.288 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:59:02.288 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:59:02.363 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:59:02.363 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:59:03.142 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:59:03.251 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:03.845 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:04.129 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T17:59:04.130 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:04.130 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:59:04.130 INFO:teuthology.orchestra.run.smithi007.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:04.130 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:04.130 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:59:04.326 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados-devel-debuginfo 2024-03-25T17:59:04.897 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:36:18 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:59:05.053 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:59:05.054 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:05.055 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:59:05.055 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:05.055 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 109 k 2024-03-25T17:59:05.055 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 440 k 2024-03-25T17:59:05.055 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:59:05.993 INFO:teuthology.orchestra.run.smithi007.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 116 kB/s | 109 kB 00:00 2024-03-25T17:59:05.994 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:05.994 INFO:teuthology.orchestra.run.smithi007.stdout:Total 116 kB/s | 109 kB 00:00 2024-03-25T17:59:05.994 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:59:05.998 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:59:05.998 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:59:06.038 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:59:06.038 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:59:06.162 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:59:06.261 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:06.636 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:06.894 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T17:59:06.895 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:06.895 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:59:06.895 INFO:teuthology.orchestra.run.smithi007.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:06.895 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:06.895 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:59:07.090 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T17:59:07.666 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:36:21 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:59:07.821 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 5.4 M 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 25 M 2024-03-25T17:59:07.822 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T17:59:24.077 INFO:teuthology.orchestra.run.smithi007.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 337 kB/s | 5.4 MB 00:16 2024-03-25T17:59:24.078 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T17:59:24.078 INFO:teuthology.orchestra.run.smithi007.stdout:Total 337 kB/s | 5.4 MB 00:16 2024-03-25T17:59:24.079 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T17:59:24.083 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T17:59:24.083 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T17:59:24.155 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T17:59:24.155 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T17:59:24.852 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T17:59:24.962 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:25.437 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:25.709 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T17:59:25.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:25.709 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T17:59:25.709 INFO:teuthology.orchestra.run.smithi007.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T17:59:25.709 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:25.710 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T17:59:25.887 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librados2-debuginfo 2024-03-25T17:59:26.461 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:36:40 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T17:59:26.615 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T17:59:26.616 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T17:59:26.617 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T17:59:26.617 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T17:59:26.617 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 44 M 2024-03-25T17:59:26.617 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 253 M 2024-03-25T17:59:26.617 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:01:54.535 INFO:teuthology.orchestra.run.smithi007.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 307 kB/s | 44 MB 02:27 2024-03-25T18:01:54.536 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:01:54.536 INFO:teuthology.orchestra.run.smithi007.stdout:Total 307 kB/s | 44 MB 02:27 2024-03-25T18:01:54.537 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:01:54.540 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:01:54.540 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:01:54.853 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:01:54.853 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:02:00.213 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:02:00.319 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:02.152 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:02:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:02:02.423 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:02:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:02:02.423 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:02:02.423 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:02:02.615 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:02:03.193 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:39:17 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:02:03.350 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:02:03.350 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 52 M 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 308 M 2024-03-25T18:02:03.351 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:03:36.499 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-osd-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 342 kB/s | 287 MB 14:19 2024-03-25T18:03:36.500 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:03:36.500 INFO:teuthology.orchestra.run.smithi183.stdout:Total 342 kB/s | 287 MB 14:19 2024-03-25T18:03:36.500 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:03:36.503 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:03:36.503 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:03:38.206 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:03:38.207 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:04:07.674 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:04:07.784 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:14.873 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:15.146 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:04:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:15.147 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:04:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-osd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:15.147 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:15.147 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:04:15.374 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install ceph-fuse-debuginfo 2024-03-25T18:04:15.941 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:41:29 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:04:16.090 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.9 M 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:16.091 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 5.9 M 2024-03-25T18:04:16.092 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 28 M 2024-03-25T18:04:16.092 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:04:27.519 INFO:teuthology.orchestra.run.smithi183.stdout:ceph-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x 533 kB/s | 5.9 MB 00:11 2024-03-25T18:04:27.520 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:27.521 INFO:teuthology.orchestra.run.smithi183.stdout:Total 532 kB/s | 5.9 MB 00:11 2024-03-25T18:04:27.521 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:04:27.525 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:04:27.525 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:04:27.597 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:04:27.597 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:04:28.336 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:04:28.443 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:28.986 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:29.239 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:04:29.239 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:29.239 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:04:29.239 INFO:teuthology.orchestra.run.smithi183.stdout: ceph-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:29.240 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:29.240 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:04:29.450 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados-devel-debuginfo 2024-03-25T18:04:30.010 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:41:44 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:04:30.163 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:04:30.163 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:30.163 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:04:30.163 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:30.163 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:04:30.163 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 109 k 2024-03-25T18:04:30.163 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:30.164 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:04:30.164 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:30.164 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:04:30.164 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:30.164 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 109 k 2024-03-25T18:04:30.164 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 440 k 2024-03-25T18:04:30.164 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:04:31.122 INFO:teuthology.orchestra.run.smithi183.stdout:librados-devel-debuginfo-19.0.0-1439.g5dcb190d. 114 kB/s | 109 kB 00:00 2024-03-25T18:04:31.122 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:31.123 INFO:teuthology.orchestra.run.smithi183.stdout:Total 113 kB/s | 109 kB 00:00 2024-03-25T18:04:31.123 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:04:31.126 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:04:31.127 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:04:31.162 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:04:31.162 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:04:31.283 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:04:31.395 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:31.790 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:32.037 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:04:32.037 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:32.037 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:04:32.037 INFO:teuthology.orchestra.run.smithi183.stdout: librados-devel-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:32.037 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:32.038 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:04:32.201 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install libcephfs2-debuginfo 2024-03-25T18:04:32.767 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:41:46 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:04:32.916 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.4 M 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 5.4 M 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 25 M 2024-03-25T18:04:32.917 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:04:50.638 INFO:teuthology.orchestra.run.smithi007.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 317 kB/s | 52 MB 02:47 2024-03-25T18:04:50.638 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:50.638 INFO:teuthology.orchestra.run.smithi007.stdout:Total 317 kB/s | 52 MB 02:47 2024-03-25T18:04:50.639 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:04:50.642 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:04:50.642 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:04:51.001 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:04:51.001 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:04:51.521 INFO:teuthology.orchestra.run.smithi183.stdout:libcephfs2-debuginfo-19.0.0-1439.g5dcb190d.el9. 295 kB/s | 5.4 MB 00:18 2024-03-25T18:04:51.522 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:04:51.523 INFO:teuthology.orchestra.run.smithi183.stdout:Total 295 kB/s | 5.4 MB 00:18 2024-03-25T18:04:51.523 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:04:51.527 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:04:51.527 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:04:51.594 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:04:51.595 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:04:52.294 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:04:52.395 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:52.931 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:53.195 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:04:53.195 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:53.195 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:04:53.195 INFO:teuthology.orchestra.run.smithi183.stdout: libcephfs2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:53.196 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:53.196 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:04:53.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librados2-debuginfo 2024-03-25T18:04:53.964 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:42:07 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:04:54.112 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 44 M 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:04:54.113 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:04:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:04:54.114 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:04:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 44 M 2024-03-25T18:04:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 253 M 2024-03-25T18:04:54.114 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:04:57.346 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:04:57.439 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:59.625 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:59.886 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:04:59.886 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:04:59.886 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:04:59.886 INFO:teuthology.orchestra.run.smithi007.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:04:59.886 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:04:59.886 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:05:00.069 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:05:00.639 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:42:14 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:05:00.794 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:05:00.795 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 615 k 2024-03-25T18:05:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.1 M 2024-03-25T18:05:00.796 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:05:02.723 INFO:teuthology.orchestra.run.smithi007.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 319 kB/s | 615 kB 00:01 2024-03-25T18:05:02.723 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:02.724 INFO:teuthology.orchestra.run.smithi007.stdout:Total 319 kB/s | 615 kB 00:01 2024-03-25T18:05:02.724 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:05:02.728 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:05:02.728 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:05:02.772 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:05:02.773 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:05:02.946 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:05:03.075 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:03.608 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:03.869 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:05:03.869 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:03.869 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:05:03.869 INFO:teuthology.orchestra.run.smithi007.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:03.869 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:03.869 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:05:04.183 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:05:04.758 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:42:18 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:05:04.912 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:05:04.913 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:04.913 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:05:04.913 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:04.913 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:05:04.913 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:05:04.913 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:05:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:05:04.914 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 771 k 2024-03-25T18:05:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.7 M 2024-03-25T18:05:04.914 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:05:08.225 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 233 kB/s | 771 kB 00:03 2024-03-25T18:05:08.226 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:08.226 INFO:teuthology.orchestra.run.smithi007.stdout:Total 233 kB/s | 771 kB 00:03 2024-03-25T18:05:08.226 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:05:08.230 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:05:08.231 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:05:08.274 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:05:08.274 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:05:08.462 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:05:08.577 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:08.951 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:09.217 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:05:09.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:05:09.980 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:42:23 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:05:10.135 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:10.136 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:05:10.137 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:10.137 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.1 M 2024-03-25T18:05:10.137 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 4.3 M 2024-03-25T18:05:10.137 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:05:13.103 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 388 kB/s | 1.1 MB 00:02 2024-03-25T18:05:13.103 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:13.104 INFO:teuthology.orchestra.run.smithi007.stdout:Total 388 kB/s | 1.1 MB 00:02 2024-03-25T18:05:13.104 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:05:13.108 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:05:13.108 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:05:13.153 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:05:13.154 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:05:13.387 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:05:13.489 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:13.921 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:14.195 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:14.195 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:14.195 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:05:14.195 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:14.195 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:14.195 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:05:14.385 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:05:14.964 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:42:28 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:05:15.120 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:05:15.120 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:05:15.121 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:15.122 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 337 k 2024-03-25T18:05:15.122 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 1.0 M 2024-03-25T18:05:15.122 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:05:16.306 INFO:teuthology.orchestra.run.smithi007.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 285 kB/s | 337 kB 00:01 2024-03-25T18:05:16.306 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:16.307 INFO:teuthology.orchestra.run.smithi007.stdout:Total 285 kB/s | 337 kB 00:01 2024-03-25T18:05:16.307 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:05:16.311 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:05:16.311 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:05:16.352 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:05:16.352 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:05:16.499 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:05:16.607 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:17.032 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:17.291 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:05:17.291 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:17.291 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:05:17.291 INFO:teuthology.orchestra.run.smithi007.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:17.291 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:17.291 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:05:17.461 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:05:18.034 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:42:32 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:05:18.189 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:05:18.190 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:18.191 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 509 k 2024-03-25T18:05:18.191 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 2.5 M 2024-03-25T18:05:18.191 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:05:20.842 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 192 kB/s | 509 kB 00:02 2024-03-25T18:05:20.843 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:05:20.843 INFO:teuthology.orchestra.run.smithi007.stdout:Total 192 kB/s | 509 kB 00:02 2024-03-25T18:05:20.843 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:05:20.847 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:05:20.847 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:05:20.888 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:05:20.889 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:05:21.056 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:05:21.169 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:21.520 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:05:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:21.792 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:05:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:05:21.792 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:21.792 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:05:21.976 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:05:22.553 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:42:36 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:05:22.709 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:05:22.709 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:22.709 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:05:22.709 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:22.709 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:05:22.709 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:05:22.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:22.710 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:05:22.710 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:05:22.710 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:05:22.710 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:05:22.710 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 52 M 2024-03-25T18:05:22.710 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 298 M 2024-03-25T18:05:22.711 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:07:14.071 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 474 kB/s | 52 MB 01:51 2024-03-25T18:07:14.071 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:14.072 INFO:teuthology.orchestra.run.smithi007.stdout:Total 474 kB/s | 52 MB 01:51 2024-03-25T18:07:14.072 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:07:14.075 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:07:14.075 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:07:14.425 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:07:14.426 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:07:20.673 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:07:20.771 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:22.903 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:23.180 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:07:23.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:07:23.180 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:07:23.180 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:23.180 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:07:23.180 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:07:23.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:07:23.938 INFO:teuthology.orchestra.run.smithi007.stdout:Last metadata expiration check: 1:44:37 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:07:24.092 INFO:teuthology.orchestra.run.smithi007.stdout:Dependencies resolved. 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout: Package Arch Version Repo Size 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout:Installing: 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction Summary 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout:================================================================================ 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout:Install 1 Package 2024-03-25T18:07:24.093 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:07:24.094 INFO:teuthology.orchestra.run.smithi007.stdout:Total download size: 1.3 M 2024-03-25T18:07:24.094 INFO:teuthology.orchestra.run.smithi007.stdout:Installed size: 5.5 M 2024-03-25T18:07:24.094 INFO:teuthology.orchestra.run.smithi007.stdout:Downloading Packages: 2024-03-25T18:07:30.105 INFO:teuthology.orchestra.run.smithi007.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 213 kB/s | 1.3 MB 00:06 2024-03-25T18:07:30.106 INFO:teuthology.orchestra.run.smithi007.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:30.106 INFO:teuthology.orchestra.run.smithi007.stdout:Total 213 kB/s | 1.3 MB 00:06 2024-03-25T18:07:30.106 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction check 2024-03-25T18:07:30.110 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction check succeeded. 2024-03-25T18:07:30.110 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction test 2024-03-25T18:07:30.157 INFO:teuthology.orchestra.run.smithi007.stdout:Transaction test succeeded. 2024-03-25T18:07:30.157 INFO:teuthology.orchestra.run.smithi007.stdout:Running transaction 2024-03-25T18:07:30.407 INFO:teuthology.orchestra.run.smithi007.stdout: Preparing : 1/1 2024-03-25T18:07:30.498 INFO:teuthology.orchestra.run.smithi007.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:30.922 INFO:teuthology.orchestra.run.smithi007.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:31.189 INFO:teuthology.orchestra.run.smithi007.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:07:31.189 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:07:31.190 INFO:teuthology.orchestra.run.smithi007.stdout:Installed: 2024-03-25T18:07:31.190 INFO:teuthology.orchestra.run.smithi007.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:31.190 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:07:31.190 INFO:teuthology.orchestra.run.smithi007.stdout:Complete! 2024-03-25T18:07:31.366 DEBUG:teuthology.parallel:result is None 2024-03-25T18:07:39.907 INFO:teuthology.orchestra.run.smithi183.stdout:librados2-debuginfo-19.0.0-1439.g5dcb190d.el9.x 274 kB/s | 44 MB 02:45 2024-03-25T18:07:39.907 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:07:39.908 INFO:teuthology.orchestra.run.smithi183.stdout:Total 274 kB/s | 44 MB 02:45 2024-03-25T18:07:39.908 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:07:39.912 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:07:39.912 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:07:40.212 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:07:40.212 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:07:45.256 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:07:45.355 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:47.130 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:47.377 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_ 1/1 2024-03-25T18:07:47.377 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:07:47.377 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:07:47.377 INFO:teuthology.orchestra.run.smithi183.stdout: librados2-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:07:47.377 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:07:47.377 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:07:47.576 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install librbd1-debuginfo 2024-03-25T18:07:48.145 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:45:02 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:07:48.297 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:07:48.298 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:07:48.298 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:07:48.298 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:07:48.298 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 52 M 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 308 M 2024-03-25T18:07:48.299 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:09:32.785 INFO:teuthology.orchestra.run.smithi183.stdout:librbd1-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 508 kB/s | 52 MB 01:44 2024-03-25T18:09:32.786 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:32.786 INFO:teuthology.orchestra.run.smithi183.stdout:Total 508 kB/s | 52 MB 01:44 2024-03-25T18:09:32.786 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:09:32.789 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:09:32.789 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:09:33.134 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:09:33.134 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:09:39.108 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:09:39.224 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:41.266 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:41.525 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:09:41.525 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:41.525 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:09:41.525 INFO:teuthology.orchestra.run.smithi183.stdout: librbd1-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:41.525 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:41.525 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:09:41.715 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-cephfs-debuginfo 2024-03-25T18:09:42.277 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:46:56 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:09:42.425 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:09:42.425 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 615 k 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 615 k 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.1 M 2024-03-25T18:09:42.426 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:09:45.053 INFO:teuthology.orchestra.run.smithi183.stdout:python3-cephfs-debuginfo-19.0.0-1439.g5dcb190d. 234 kB/s | 615 kB 00:02 2024-03-25T18:09:45.053 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:45.054 INFO:teuthology.orchestra.run.smithi183.stdout:Total 234 kB/s | 615 kB 00:02 2024-03-25T18:09:45.054 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:09:45.058 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:09:45.058 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:09:45.098 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:09:45.098 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:09:45.260 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:09:45.344 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:09:45.739 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:09:45.991 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9 1/1 2024-03-25T18:09:45.992 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:45.992 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:09:45.992 INFO:teuthology.orchestra.run.smithi183.stdout: python3-cephfs-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:45.992 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:45.992 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:09:46.177 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rados-debuginfo 2024-03-25T18:09:46.743 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:47:00 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:09:46.893 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:09:46.893 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 771 k 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 771 k 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.7 M 2024-03-25T18:09:46.894 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:09:52.680 INFO:teuthology.orchestra.run.smithi183.stdout:python3-rados-debuginfo-19.0.0-1439.g5dcb190d.e 133 kB/s | 771 kB 00:05 2024-03-25T18:09:52.680 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:52.681 INFO:teuthology.orchestra.run.smithi183.stdout:Total 133 kB/s | 771 kB 00:05 2024-03-25T18:09:52.681 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:09:52.685 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:09:52.685 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:09:52.727 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:09:52.727 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:09:52.910 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:09:53.013 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:09:53.399 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:09:53.650 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9. 1/1 2024-03-25T18:09:53.650 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:53.650 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:09:53.650 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rados-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:53.650 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:53.650 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:09:53.863 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rbd-debuginfo 2024-03-25T18:09:54.429 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:47:08 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:09:54.578 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:09:54.578 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.1 M 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 1.1 M 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 4.3 M 2024-03-25T18:09:54.579 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:09:57.800 INFO:teuthology.orchestra.run.smithi183.stdout:python3-rbd-debuginfo-19.0.0-1439.g5dcb190d.el9 357 kB/s | 1.1 MB 00:03 2024-03-25T18:09:57.800 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:09:57.800 INFO:teuthology.orchestra.run.smithi183.stdout:Total 357 kB/s | 1.1 MB 00:03 2024-03-25T18:09:57.800 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:09:57.804 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:09:57.804 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:09:57.846 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:09:57.847 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:09:58.071 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:09:58.189 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:09:58.607 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:09:58.856 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:09:58.856 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:58.856 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:09:58.856 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:09:58.856 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:58.857 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:09:59.070 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install python3-rgw-debuginfo 2024-03-25T18:09:59.636 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:47:13 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:09:59.788 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:09:59.788 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:59.788 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:09:59.788 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 337 k 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:09:59.789 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 337 k 2024-03-25T18:09:59.790 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 1.0 M 2024-03-25T18:09:59.790 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:10:02.059 INFO:teuthology.orchestra.run.smithi183.stdout:python3-rgw-debuginfo-19.0.0-1439.g5dcb190d.el9 149 kB/s | 337 kB 00:02 2024-03-25T18:10:02.060 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:02.061 INFO:teuthology.orchestra.run.smithi183.stdout:Total 149 kB/s | 337 kB 00:02 2024-03-25T18:10:02.061 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:10:02.065 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:10:02.065 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:10:02.103 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:10:02.104 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:10:02.239 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:10:02.341 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:02.693 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:02.945 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x8 1/1 2024-03-25T18:10:02.946 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:02.946 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:10:02.946 INFO:teuthology.orchestra.run.smithi183.stdout: python3-rgw-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:02.946 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:02.946 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:10:03.144 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-fuse-debuginfo 2024-03-25T18:10:03.711 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:47:17 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:10:03.859 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 509 k 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:10:03.860 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:10:03.861 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:03.861 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 509 k 2024-03-25T18:10:03.861 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 2.5 M 2024-03-25T18:10:03.861 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:10:08.171 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-fuse-debuginfo-19.0.0-1439.g5dcb190d.el9.x8 118 kB/s | 509 kB 00:04 2024-03-25T18:10:08.171 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:10:08.172 INFO:teuthology.orchestra.run.smithi183.stdout:Total 118 kB/s | 509 kB 00:04 2024-03-25T18:10:08.172 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:10:08.176 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:10:08.176 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:10:08.215 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:10:08.215 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:10:08.379 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:10:08.485 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:08.901 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:09.155 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_6 1/1 2024-03-25T18:10:09.155 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:09.155 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:10:09.155 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-fuse-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:10:09.155 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:09.156 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:10:09.351 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-mirror-debuginfo 2024-03-25T18:10:09.909 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:47:23 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:10:10.057 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:10:10.057 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 52 M 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:10:10.058 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:10:10.059 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 52 M 2024-03-25T18:10:10.059 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 298 M 2024-03-25T18:10:10.059 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:13:03.705 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-mirror-debuginfo-19.0.0-1439.g5dcb190d.el9. 304 kB/s | 52 MB 02:53 2024-03-25T18:13:03.706 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:03.706 INFO:teuthology.orchestra.run.smithi183.stdout:Total 304 kB/s | 52 MB 02:53 2024-03-25T18:13:03.707 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:13:03.709 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:13:03.709 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:13:04.045 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:13:04.046 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:13:09.949 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:13:10.052 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:12.013 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:12.262 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86 1/1 2024-03-25T18:13:12.262 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:12.262 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:13:12.262 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-mirror-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:12.263 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:12.263 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:13:12.452 DEBUG:teuthology.orchestra.run.smithi183:> sudo yum -y install rbd-nbd-debuginfo 2024-03-25T18:13:13.019 INFO:teuthology.orchestra.run.smithi183.stdout:Last metadata expiration check: 1:50:27 ago on Mon 25 Mar 2024 04:22:46 PM UTC. 2024-03-25T18:13:13.167 INFO:teuthology.orchestra.run.smithi183.stdout:Dependencies resolved. 2024-03-25T18:13:13.168 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:13:13.168 INFO:teuthology.orchestra.run.smithi183.stdout: Package Arch Version Repo Size 2024-03-25T18:13:13.168 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:13:13.168 INFO:teuthology.orchestra.run.smithi183.stdout:Installing: 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-debuginfo x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.3 M 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction Summary 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout:================================================================================ 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout:Install 1 Package 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout:Total download size: 1.3 M 2024-03-25T18:13:13.169 INFO:teuthology.orchestra.run.smithi183.stdout:Installed size: 5.5 M 2024-03-25T18:13:13.170 INFO:teuthology.orchestra.run.smithi183.stdout:Downloading Packages: 2024-03-25T18:13:17.947 INFO:teuthology.orchestra.run.smithi183.stdout:rbd-nbd-debuginfo-19.0.0-1439.g5dcb190d.el9.x86 268 kB/s | 1.3 MB 00:04 2024-03-25T18:13:17.948 INFO:teuthology.orchestra.run.smithi183.stdout:-------------------------------------------------------------------------------- 2024-03-25T18:13:17.948 INFO:teuthology.orchestra.run.smithi183.stdout:Total 268 kB/s | 1.3 MB 00:04 2024-03-25T18:13:17.949 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction check 2024-03-25T18:13:17.952 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction check succeeded. 2024-03-25T18:13:17.952 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction test 2024-03-25T18:13:17.996 INFO:teuthology.orchestra.run.smithi183.stdout:Transaction test succeeded. 2024-03-25T18:13:17.997 INFO:teuthology.orchestra.run.smithi183.stdout:Running transaction 2024-03-25T18:13:18.238 INFO:teuthology.orchestra.run.smithi183.stdout: Preparing : 1/1 2024-03-25T18:13:18.329 INFO:teuthology.orchestra.run.smithi183.stdout: Installing : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:18.733 INFO:teuthology.orchestra.run.smithi183.stdout: Running scriptlet: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: Verifying : rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-25T18:13:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:18.996 INFO:teuthology.orchestra.run.smithi183.stdout:Installed: 2024-03-25T18:13:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: rbd-nbd-debuginfo-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-25T18:13:18.996 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:18.996 INFO:teuthology.orchestra.run.smithi183.stdout:Complete! 2024-03-25T18:13:19.201 DEBUG:teuthology.parallel:result is None 2024-03-25T18:13:19.201 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:13:19.420 DEBUG:teuthology.orchestra.run.smithi007:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:13:19.444 INFO:teuthology.orchestra.run.smithi007.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:19.445 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:19.445 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:13:19.447 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-25T18:13:19.587 DEBUG:teuthology.orchestra.run.smithi183:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-25T18:13:19.607 INFO:teuthology.orchestra.run.smithi183.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:19.607 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-25T18:13:19.607 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-25T18:13:19.609 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-25T18:13:19.609 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:19.609 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:13:19.641 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:19.641 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-25T18:13:19.674 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-25T18:13:19.674 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:19.674 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:13:19.713 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:13:19.782 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:19.782 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/daemon-helper 2024-03-25T18:13:19.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-25T18:13:19.875 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-25T18:13:19.875 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:19.875 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:13:19.905 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:13:19.973 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:19.973 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-25T18:13:20.000 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-25T18:13:20.066 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-25T18:13:20.066 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:20.066 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:13:20.095 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:13:20.164 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:20.164 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/usr/bin/stdin-killer 2024-03-25T18:13:20.190 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-25T18:13:20.257 INFO:teuthology.run_tasks:Running task ceph... 2024-03-25T18:13:20.332 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-25T18:13:20.332 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 777 /var/log/ceph 2024-03-25T18:13:20.334 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 777 /var/log/ceph 2024-03-25T18:13:20.361 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-25T18:13:20.361 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:13:20.404 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-25T18:13:20.430 INFO:tasks.ceph:Creating extra log directories... 2024-03-25T18:13:20.430 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:13:20.473 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-25T18:13:20.526 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-25T18:13:20.527 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-25T18:13:20.527 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind}', 'email': None, 'first_in_suite': False, 'job_id': '7621262', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'but it is still running', 'objects unfound and apparently lost', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request', 'overall HEALTH_', '\\(PG_', '\\(MON_DOWN\\)', '\\(SLOW_OPS\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'valgrind': {'mds': ['--tool=memcheck'], 'mon': ['--tool=memcheck', '--leak-check=full', '--show-reachable=yes'], 'osd': ['--tool=memcheck']}}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'install': {'ceph': {'debuginfo': True, 'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 6785, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi007.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNO605m0K+s3ytSEjhPuzWOnVvx5mM9qJa4VSjBMg+mUch9L3YOnpZK3XO1/G98YOVP8dPUsCh76nbIdEPo2i1s=', 'smithi183.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCCm4RjM8duBU3f188Ut1weiOsvtzl+k29zkv296jFRM9XUdjgElzX0FBFJ83grLAIK4EwEXwcZIeVDmVJFqeMM='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'thrashosds': {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200}}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-25_13:28:33', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-25T18:13:20.527 DEBUG:teuthology.orchestra.run.smithi007:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:13:20.542 DEBUG:teuthology.orchestra.run.smithi183:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-25T18:13:20.556 DEBUG:teuthology.orchestra.run.smithi007:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:13:20.596 DEBUG:teuthology.orchestra.run.smithi183:> sudo install -d -m0777 -- /var/run/ceph 2024-03-25T18:13:20.623 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:20.623 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:13:20.677 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:13:20.677 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_1 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-25 16:27:16.766485426 +0000 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-25 16:22:01.543738996 +0000 2024-03-25T18:13:20.732 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-25 16:22:01.543738996 +0000 2024-03-25T18:13:20.733 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-25 16:22:01.543738996 +0000 2024-03-25T18:13:20.733 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:13:20.798 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-25T18:13:20.798 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-25T18:13:20.798 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 0.00010065 s, 5.1 MB/s 2024-03-25T18:13:20.799 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:13:20.855 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_2 2024-03-25T18:13:20.909 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-25 16:27:16.767485410 +0000 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-25 16:22:01.878733779 +0000 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-25 16:22:01.878733779 +0000 2024-03-25T18:13:20.910 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-25 16:22:01.878733779 +0000 2024-03-25T18:13:20.910 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:13:20.974 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-25T18:13:20.974 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-25T18:13:20.974 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 7.6583e-05 s, 6.7 MB/s 2024-03-25T18:13:20.976 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:13:21.030 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_3 2024-03-25T18:13:21.084 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-25 16:27:16.767485410 +0000 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-25 16:22:02.217728499 +0000 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-25 16:22:02.217728499 +0000 2024-03-25T18:13:21.085 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-25 16:22:02.217728499 +0000 2024-03-25T18:13:21.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:13:21.149 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-25T18:13:21.149 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-25T18:13:21.150 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 8.2111e-05 s, 6.2 MB/s 2024-03-25T18:13:21.150 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:13:21.205 DEBUG:teuthology.orchestra.run.smithi007:> stat /dev/vg_nvme/lv_4 2024-03-25T18:13:21.259 INFO:teuthology.orchestra.run.smithi007.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:13:21.259 INFO:teuthology.orchestra.run.smithi007.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:21.260 INFO:teuthology.orchestra.run.smithi007.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-25T18:13:21.260 INFO:teuthology.orchestra.run.smithi007.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:21.260 INFO:teuthology.orchestra.run.smithi007.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:21.260 INFO:teuthology.orchestra.run.smithi007.stdout:Access: 2024-03-25 16:27:16.767485410 +0000 2024-03-25T18:13:21.260 INFO:teuthology.orchestra.run.smithi007.stdout:Modify: 2024-03-25 16:22:02.553723266 +0000 2024-03-25T18:13:21.260 INFO:teuthology.orchestra.run.smithi007.stdout:Change: 2024-03-25 16:22:02.553723266 +0000 2024-03-25T18:13:21.260 INFO:teuthology.orchestra.run.smithi007.stdout: Birth: 2024-03-25 16:22:02.553723266 +0000 2024-03-25T18:13:21.260 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:13:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records in 2024-03-25T18:13:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:1+0 records out 2024-03-25T18:13:21.324 INFO:teuthology.orchestra.run.smithi007.stderr:512 bytes copied, 9.4216e-05 s, 5.4 MB/s 2024-03-25T18:13:21.325 DEBUG:teuthology.orchestra.run.smithi007:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:13:21.380 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:21.381 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:21.381 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/scratch_devs of=/dev/stdout 2024-03-25T18:13:21.398 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-25T18:13:21.398 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_1 2024-03-25T18:13:21.455 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-25T18:13:21.455 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:21.455 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-25T18:13:21.455 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:21.455 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:21.455 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-25 16:27:46.085904901 +0000 2024-03-25T18:13:21.456 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-25 16:21:59.858623401 +0000 2024-03-25T18:13:21.456 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-25 16:21:59.858623401 +0000 2024-03-25T18:13:21.456 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-25 16:21:59.858623401 +0000 2024-03-25T18:13:21.456 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-25T18:13:21.523 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-25T18:13:21.523 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-25T18:13:21.523 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000198334 s, 2.6 MB/s 2024-03-25T18:13:21.525 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-25T18:13:21.582 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_2 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 687 Links: 1 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-25 16:27:46.085904901 +0000 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-25 16:22:00.193614323 +0000 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-25 16:22:00.193614323 +0000 2024-03-25T18:13:21.638 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-25 16:22:00.193614323 +0000 2024-03-25T18:13:21.638 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-25T18:13:21.705 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-25T18:13:21.705 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-25T18:13:21.705 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.0001968 s, 2.6 MB/s 2024-03-25T18:13:21.706 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-25T18:13:21.763 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_3 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-25 16:27:46.085904901 +0000 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-25 16:22:00.530605191 +0000 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-25 16:22:00.530605191 +0000 2024-03-25T18:13:21.819 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-25 16:22:00.530605191 +0000 2024-03-25T18:13:21.820 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-25T18:13:21.886 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-25T18:13:21.886 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-25T18:13:21.886 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000195649 s, 2.6 MB/s 2024-03-25T18:13:21.887 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-25T18:13:21.943 DEBUG:teuthology.orchestra.run.smithi183:> stat /dev/vg_nvme/lv_4 2024-03-25T18:13:21.998 INFO:teuthology.orchestra.run.smithi183.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout:Context: system_u:object_r:device_t:s0 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout:Access: 2024-03-25 16:27:46.086904873 +0000 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout:Modify: 2024-03-25 16:22:00.854596411 +0000 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout:Change: 2024-03-25 16:22:00.854596411 +0000 2024-03-25T18:13:21.999 INFO:teuthology.orchestra.run.smithi183.stdout: Birth: 2024-03-25 16:22:00.854596411 +0000 2024-03-25T18:13:21.999 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-25T18:13:22.064 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records in 2024-03-25T18:13:22.064 INFO:teuthology.orchestra.run.smithi183.stderr:1+0 records out 2024-03-25T18:13:22.065 INFO:teuthology.orchestra.run.smithi183.stderr:512 bytes copied, 0.000175761 s, 2.9 MB/s 2024-03-25T18:13:22.065 DEBUG:teuthology.orchestra.run.smithi183:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-25T18:13:22.121 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:22.121 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:13:22.121 INFO:tasks.ceph:Generating config... 2024-03-25T18:13:22.122 INFO:tasks.ceph:[client] debug monc = 20 2024-03-25T18:13:22.123 INFO:tasks.ceph:[client] debug ms = 1 2024-03-25T18:13:22.123 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-25T18:13:22.123 INFO:tasks.ceph:[client] debug rados = 20 2024-03-25T18:13:22.123 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-25T18:13:22.123 INFO:tasks.ceph:[global] mon election default strategy = 3 2024-03-25T18:13:22.123 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-25T18:13:22.123 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-25T18:13:22.123 INFO:tasks.ceph:[global] ms type = async 2024-03-25T18:13:22.123 INFO:tasks.ceph:[global] osd heartbeat grace = 80 2024-03-25T18:13:22.123 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-25T18:13:22.123 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-25T18:13:22.123 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-25T18:13:22.123 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-25T18:13:22.123 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-25T18:13:22.123 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-25T18:13:22.123 INFO:tasks.ceph:[mon] mon osd crush smoke test = False 2024-03-25T18:13:22.124 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-25T18:13:22.124 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-25T18:13:22.124 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-25T18:13:22.124 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-25T18:13:22.124 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-25T18:13:22.124 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] bdev async discard = True 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] bluestore allocator = stupid 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] debug bluefs = 1 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] debug bluestore = 1 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-25T18:13:22.124 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd fast shutdown = False 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-25T18:13:22.125 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-25T18:13:22.126 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-25T18:13:22.126 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-25T18:13:22.126 INFO:tasks.ceph:Setting up mon.a... 2024-03-25T18:13:22.126 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:22.204 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.keyring 2024-03-25T18:13:22.206 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-25T18:13:22.254 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:22.282 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.7:6789]'), ('mon.c', '[v1:172.21.15.7:6790]'), ('mon.b', '[v1:172.21.15.183:6789]')] 2024-03-25T18:13:22.283 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.7:6789],[v1:172.21.15.7:6790],[v1:172.21.15.183:6789]', 'mon client directed command retry': 5, 'mon election default strategy': 3, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async', 'osd heartbeat grace': 80}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'stupid', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 1, 'debug bluestore': 1, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd fast shutdown': False, 'osd max markdown count': 1000, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osd crush smoke test': False, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-25T18:13:22.284 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:22.284 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:13:22.337 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.7:6789]' --addv c '[v1:172.21.15.7:6790]' --addv b '[v1:172.21.15.183:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:22.422 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: generated fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:setting min_mon_release = quincy 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:epoch 0 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:min_mon_release 17 (quincy) 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:election_strategy: 3 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:13:22.423 INFO:teuthology.orchestra.run.smithi007.stdout:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:13:22.424 INFO:teuthology.orchestra.run.smithi007.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-25T18:13:22.425 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-25T18:13:22.439 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 81cdc36e-13e4-4148-a153-7df7e1b15d1a... 2024-03-25T18:13:22.440 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:13:22.482 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-25T18:13:22.529 INFO:teuthology.orchestra.run.smithi007.stdout:[global] 2024-03-25T18:13:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: chdir = "" 2024-03-25T18:13:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:13:22.529 INFO:teuthology.orchestra.run.smithi007.stdout: auth supported = cephx 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: filestore xattr use omap = true 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: mon clock drift allowed = 1.000 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: osd crush chooseleaf type = 0 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: auth debug = true 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on old message = true 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: ms die on bug = true 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: mon pg warn max object skew = 0 2024-03-25T18:13:22.530 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default size = 2 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow primary affinity = true 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd allow pg remap = true 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on no sortbitwise = false 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on too few osds = false 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: mon_allow_pool_size_one = true 2024-03-25T18:13:22.531 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: osd default data pool replay window = 5 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon allow pool delete = true 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon cluster log file level = debug 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: debug asserts on shutdown = true 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon health detail to clog = false 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon host = "[v1:172.21.15.7:6789],[v1:172.21.15.7:6790],[v1:172.21.15.183:6789]" 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon client directed command retry = 5 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: mon election default strategy = 3 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: ms bind msgr2 = False 2024-03-25T18:13:22.532 INFO:teuthology.orchestra.run.smithi007.stdout: ms inject socket failures = 5000 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: ms type = async 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd heartbeat grace = 80 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: fsid = 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout:[osd] 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd journal size = 100 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub load threshold = 5.0 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd scrub max interval = 600 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd recover clone overlap = true 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd recovery max chunk = 1048576 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug shutdown = true 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug op order = true 2024-03-25T18:13:22.533 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify stray on activate = true 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug trim objects = true 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: osd open classes on start = true 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug pg log writeout = true 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: osd map max advance = 10 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: journal zero on create = true 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: filestore apply finisher threads = 3 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.534 INFO:teuthology.orchestra.run.smithi007.stdout: bdev debug aio = true 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug misdirected ops = true 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: bdev async discard = True 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: bdev enable discard = True 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore allocator = stupid 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore block size = 96636764160 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore fsck on mount = True 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: bluestore zero block detection = True 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluefs = 1 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: debug bluestore = 1 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: debug osd = 20 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: debug rocksdb = 10 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:22.535 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd full ratio = 0.9 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd class default list = * 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd class load list = * 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd client watch timeout = 120 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify cached snaps = True 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd debug verify missing on start = True 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd fast shutdown = False 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd max markdown count = 1000 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd mclock override recovery settings = True 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd objectstore = bluestore 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue = debug_random 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd op queue cut off = debug_random 2024-03-25T18:13:22.536 INFO:teuthology.orchestra.run.smithi007.stdout: osd shutdown pgref assert = True 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr] 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug mgr = 20 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: mgr/telemetry/nag = false 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout:[mon] 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug mon = 20 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug paxos = 20 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: debug auth = 20 2024-03-25T18:13:22.537 INFO:teuthology.orchestra.run.smithi007.stdout: mon data avail warn = 5 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: mon mgr mkfs grace = 240 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd reporter subtree level = osd 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd prime pg temp = true 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.538 INFO:teuthology.orchestra.run.smithi007.stdout: # 1m isn't quite enough 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon min osdmap epochs = 50 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon osd crush smoke test = False 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune interval = 2 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune min = 15 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon scrub interval = 300 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: mon warn on pool no app = False 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: paxos service trim min = 10 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout:[client] 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: rgw cache enabled = true 2024-03-25T18:13:22.539 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable ops log = true 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: rgw enable usage log = true 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: debug monc = 20 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: debug ms = 1 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: debug objecter = 20 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout: debug rados = 20 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.a] 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.c] 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi007.stdout:[mon.b] 2024-03-25T18:13:22.540 INFO:teuthology.orchestra.run.smithi183.stdout:[global] 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: chdir = "" 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: auth supported = cephx 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: filestore xattr use omap = true 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: mon clock drift allowed = 1.000 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: osd crush chooseleaf type = 0 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: auth debug = true 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on old message = true 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: ms die on bug = true 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.541 INFO:teuthology.orchestra.run.smithi183.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon pg warn max object skew = 0 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: # disable pg_autoscaler by default for new pools 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default size = 2 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow primary affinity = true 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd allow pg remap = true 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on legacy crush tunables = false 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on crush straw calc version zero = false 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on no sortbitwise = false 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on osd down out interval zero = false 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on too few osds = false 2024-03-25T18:13:22.542 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon_allow_pool_size_one = true 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: osd default data pool replay window = 5 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon allow pool delete = true 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon cluster log file level = debug 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: debug asserts on shutdown = true 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon health detail to clog = false 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon host = "[v1:172.21.15.7:6789],[v1:172.21.15.7:6790],[v1:172.21.15.183:6789]" 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon client directed command retry = 5 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: mon election default strategy = 3 2024-03-25T18:13:22.543 INFO:teuthology.orchestra.run.smithi183.stdout: ms bind msgr2 = False 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: ms inject socket failures = 5000 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: ms type = async 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd heartbeat grace = 80 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: fsid = 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout:[osd] 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd journal size = 100 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub load threshold = 5.0 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd scrub max interval = 600 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock profile = high_recovery_ops 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd recover clone overlap = true 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd recovery max chunk = 1048576 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug shutdown = true 2024-03-25T18:13:22.544 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug op order = true 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify stray on activate = true 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug trim objects = true 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: osd open classes on start = true 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug pg log writeout = true 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: osd deep scrub update digest min age = 30 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: osd map max advance = 10 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: journal zero on create = true 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: filestore ondisk finisher threads = 3 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: filestore apply finisher threads = 3 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.545 INFO:teuthology.orchestra.run.smithi183.stdout: bdev debug aio = true 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug misdirected ops = true 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: bdev async discard = True 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: bdev enable discard = True 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore allocator = stupid 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore block size = 96636764160 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore fsck on mount = True 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: bluestore zero block detection = True 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluefs = 1 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: debug bluestore = 1 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: debug osd = 20 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: debug rocksdb = 10 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd backfillfull_ratio = 0.85 2024-03-25T18:13:22.546 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd full ratio = 0.9 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd nearfull ratio = 0.8 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd blocked scrub grace period = 3600 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd class default list = * 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd class load list = * 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd client watch timeout = 120 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify cached snaps = True 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd debug verify missing on start = True 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd failsafe full ratio = 0.95 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd fast shutdown = False 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd max markdown count = 1000 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd mclock override recovery settings = True 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd objectstore = bluestore 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue = debug_random 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd op queue cut off = debug_random 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: osd shutdown pgref assert = True 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.547 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr] 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug mgr = 20 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: mgr/telemetry/nag = false 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout:[mon] 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug mon = 20 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug paxos = 20 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: debug auth = 20 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: mon data avail warn = 5 2024-03-25T18:13:22.548 INFO:teuthology.orchestra.run.smithi183.stdout: mon mgr mkfs grace = 240 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min pgs per osd = 4 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd reporter subtree level = osd 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd prime pg temp = true 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon reweight min bytes per osd = 10 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: auth mon ticket ttl = 660 # 11m 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: auth service ticket ttl = 240 # 4m 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: # don't complain about insecure global_id in the test suite 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: # 1m isn't quite enough 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon_down_mkfs_grace = 2m 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.549 INFO:teuthology.orchestra.run.smithi183.stdout: mon_warn_on_filestore_osds = false 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: mon min osdmap epochs = 50 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: mon osd crush smoke test = False 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune interval = 2 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune min = 15 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: mon osdmap full prune txsize = 2 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: mon scrub interval = 300 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: mon warn on pool no app = False 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: paxos service trim min = 10 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout:[client] 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: rgw cache enabled = true 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable ops log = true 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: rgw enable usage log = true 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-25T18:13:22.550 INFO:teuthology.orchestra.run.smithi183.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-25T18:13:22.551 INFO:teuthology.orchestra.run.smithi183.stdout: debug monc = 20 2024-03-25T18:13:22.551 INFO:teuthology.orchestra.run.smithi183.stdout: debug ms = 1 2024-03-25T18:13:22.551 INFO:teuthology.orchestra.run.smithi183.stdout: debug objecter = 20 2024-03-25T18:13:22.551 INFO:teuthology.orchestra.run.smithi183.stdout: debug rados = 20 2024-03-25T18:13:22.551 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.a] 2024-03-25T18:13:22.551 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.c] 2024-03-25T18:13:22.551 INFO:teuthology.orchestra.run.smithi183.stdout:[mon.b] 2024-03-25T18:13:22.552 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-25T18:13:22.552 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-25T18:13:22.607 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-25T18:13:22.611 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:22.612 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-25T18:13:22.625 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:22.625 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-25T18:13:22.679 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:13:22.679 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:22.679 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:13:22.680 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:22.757 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:22.758 DEBUG:teuthology.orchestra.run.smithi007:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:22.812 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:13:22.812 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:22.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-25T18:13:22.812 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-25T18:13:22.851 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:22.851 DEBUG:teuthology.orchestra.run.smithi183:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:22.906 INFO:tasks.ceph:Setting up mon nodes... 2024-03-25T18:13:22.906 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-25T18:13:22.906 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:13:22.960 INFO:teuthology.orchestra.run.smithi007.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-25T18:13:22.962 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:13:23.022 INFO:teuthology.orchestra.run.smithi183.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-25T18:13:23.025 INFO:tasks.ceph:Setting up mds nodes... 2024-03-25T18:13:23.025 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-25T18:13:23.025 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-25T18:13:23.067 INFO:teuthology.orchestra.run.smithi007.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-25T18:13:23.082 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-25T18:13:23.123 INFO:teuthology.orchestra.run.smithi183.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-25T18:13:23.139 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-25T18:13:23.139 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi007.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi183.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-25T18:13:23.139 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:23.165 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:23.165 INFO:tasks.ceph:role: osd.0 2024-03-25T18:13:23.166 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:13:23.166 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:13:23.231 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:23.231 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:23.232 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:23.263 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-25T18:13:23.265 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-25T18:13:23.265 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:23.337 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:23.407 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:23.474 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:23.474 INFO:tasks.ceph:role: osd.1 2024-03-25T18:13:23.474 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:13:23.474 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:13:23.540 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:23.540 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:23.540 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:23.540 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:23.541 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:23.541 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:23.541 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:23.541 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:23.541 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:23.541 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:23.572 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-25T18:13:23.574 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-25T18:13:23.575 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:23.649 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:23.720 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:23.788 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:23.788 INFO:tasks.ceph:role: osd.2 2024-03-25T18:13:23.788 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:13:23.788 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:13:23.854 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:23.854 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:23.854 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:23.855 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:23.855 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:23.887 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-25T18:13:23.889 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-25T18:13:23.889 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:23.966 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:24.039 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:24.107 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:24.107 INFO:tasks.ceph:role: osd.3 2024-03-25T18:13:24.107 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:13:24.107 DEBUG:teuthology.orchestra.run.smithi007:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:24.174 INFO:teuthology.orchestra.run.smithi007.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:24.206 INFO:teuthology.orchestra.run.smithi007.stdout:Discarding blocks...Done. 2024-03-25T18:13:24.208 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi007.front.sepia.ceph.com -o noatime 2024-03-25T18:13:24.208 DEBUG:teuthology.orchestra.run.smithi007:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:24.279 DEBUG:teuthology.orchestra.run.smithi007:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:24.350 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:24.441+0000 7fa55e013640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-25T18:13:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:24.442+0000 7fa55e013640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-25T18:13:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:24.442+0000 7fa55e013640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:24.442+0000 7fa55e013640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:24.442+0000 7fa55e013640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-25T18:13:24.443 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:24.442+0000 7fa55e013640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-25T18:13:27.286 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-25T18:13:27.317 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:27.406 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:27.405+0000 7fccae2c4640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-25T18:13:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:27.406+0000 7fccae2c4640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-25T18:13:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:27.406+0000 7fccae2c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:27.406+0000 7fccae2c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:27.406+0000 7fccae2c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-25T18:13:27.407 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:27.406+0000 7fccae2c4640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-25T18:13:30.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-25T18:13:30.290 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:30.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:30.379+0000 7f6bbe00d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-25T18:13:30.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:30.379+0000 7f6bbe00d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-25T18:13:30.380 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:30.379+0000 7f6bbe00d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:30.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:30.379+0000 7f6bbe00d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:30.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:30.379+0000 7f6bbe00d640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-25T18:13:30.381 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:30.380+0000 7f6bbe00d640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-25T18:13:33.215 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-25T18:13:33.245 DEBUG:teuthology.orchestra.run.smithi007:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:33.335+0000 7f1a04601640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-25T18:13:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:33.335+0000 7f1a04601640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-25T18:13:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:33.335+0000 7f1a04601640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:33.337 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:33.336+0000 7f1a04601640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:33.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:33.336+0000 7f1a04601640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-25T18:13:33.338 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:33.336+0000 7f1a04601640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-25T18:13:36.175 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-25T18:13:36.205 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:36.237 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:36.237 INFO:tasks.ceph:role: osd.4 2024-03-25T18:13:36.237 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:13:36.237 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-25T18:13:36.314 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:36.314 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:36.314 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:36.314 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:36.315 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:36.315 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:36.315 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:36.315 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:36.315 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:36.315 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:36.831 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-25T18:13:36.833 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-25T18:13:36.833 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:36.977 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:37.013 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:37.081 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:37.081 INFO:tasks.ceph:role: osd.5 2024-03-25T18:13:37.081 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:13:37.081 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-25T18:13:37.156 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:37.157 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:37.601 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-25T18:13:37.603 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-25T18:13:37.603 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:37.701 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:37.733 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:37.802 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:37.802 INFO:tasks.ceph:role: osd.6 2024-03-25T18:13:37.802 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:13:37.802 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:37.877 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:38.374 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-25T18:13:38.376 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-25T18:13:38.376 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:38.499 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:38.533 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:38.603 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-25T18:13:38.603 INFO:tasks.ceph:role: osd.7 2024-03-25T18:13:38.603 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:13:38.603 DEBUG:teuthology.orchestra.run.smithi183:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-25T18:13:38.679 INFO:teuthology.orchestra.run.smithi183.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-25T18:13:38.679 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: = sunit=0 swidth=0 blks 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-25T18:13:38.680 INFO:teuthology.orchestra.run.smithi183.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-25T18:13:39.193 INFO:teuthology.orchestra.run.smithi183.stdout:Discarding blocks...Done. 2024-03-25T18:13:39.195 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi183.front.sepia.ceph.com -o noatime 2024-03-25T18:13:39.195 DEBUG:teuthology.orchestra.run.smithi183:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:39.333 DEBUG:teuthology.orchestra.run.smithi183:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:39.369 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:39.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:39.476+0000 7f03dae08640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-25T18:13:39.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:39.477+0000 7f03dae08640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-25T18:13:39.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:39.477+0000 7f03dae08640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:39.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:39.477+0000 7f03dae08640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:39.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:39.477+0000 7f03dae08640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-25T18:13:39.478 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:39.477+0000 7f03dae08640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-25T18:13:42.335 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-25T18:13:42.365 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:42.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:42.456+0000 7f9b9380e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-25T18:13:42.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:42.456+0000 7f9b9380e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-25T18:13:42.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:42.456+0000 7f9b9380e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:42.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:42.456+0000 7f9b9380e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:42.457 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:42.456+0000 7f9b9380e640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-25T18:13:42.458 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:42.457+0000 7f9b9380e640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-25T18:13:45.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-25T18:13:45.330 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:45.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:45.418+0000 7f2c4520d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-25T18:13:45.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:45.418+0000 7f2c4520d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-25T18:13:45.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:45.418+0000 7f2c4520d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:45.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:45.418+0000 7f2c4520d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:45.419 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:45.418+0000 7f2c4520d640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-25T18:13:45.420 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:45.419+0000 7f2c4520d640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-25T18:13:48.263 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-25T18:13:48.294 DEBUG:teuthology.orchestra.run.smithi183:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:48.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:48.384+0000 7f6603b8c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-25T18:13:48.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:48.384+0000 7f6603b8c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-25T18:13:48.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:48.384+0000 7f6603b8c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:48.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:48.384+0000 7f6603b8c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:48.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:48.384+0000 7f6603b8c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-25T18:13:48.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:13:48.385+0000 7f6603b8c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-25T18:13:51.226 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-25T18:13:51.256 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-25T18:13:51.256 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:51.256 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-25T18:13:51.288 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:51.288 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-25T18:13:51.358 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:51.358 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-25T18:13:51.427 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:51.427 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-25T18:13:51.495 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:51.495 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-25T18:13:51.564 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:51.564 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-25T18:13:51.595 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:51.595 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-25T18:13:51.665 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:51.666 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-25T18:13:51.734 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:51.735 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-25T18:13:51.802 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:51.802 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-25T18:13:51.870 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:13:51.870 DEBUG:teuthology.orchestra.run.smithi007:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-25T18:13:51.887 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:13:51.887 DEBUG:teuthology.orchestra.run.smithi183:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-25T18:13:51.925 INFO:tasks.ceph:Adding keys to all mons... 2024-03-25T18:13:51.925 DEBUG:teuthology.orchestra.run.smithi007:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:13:51.930 DEBUG:teuthology.orchestra.run.smithi183:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-25T18:13:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.y] 2024-03-25T18:13:51.968 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDCvgFm4VxAORAAq1hyBXq/wGAYljsMb5yT6w== 2024-03-25T18:13:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.0] 2024-03-25T18:13:51.968 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDEvgFmbf1lGhAAjLMj6gm8Zd78B5uR2XPC8Q== 2024-03-25T18:13:51.968 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.1] 2024-03-25T18:13:51.968 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDHvgFmw7ZAGBAAOQZ+wP5WMH+ioVXfpHaMPg== 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.2] 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDKvgFmtcquFhAAoPEGDzb+383//k5A9gaXxA== 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.3] 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDNvgFmUm4SFBAA0W/NwBGxcDyyI66ogpaAwg== 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:[mgr.x] 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDDvgFm0wpgARAAASsumoiOoUSW/gKCuPcmeA== 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.4] 2024-03-25T18:13:51.969 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDTvgFm4u97HBAAGcwJA2m7Alfh6CcqBIgdwg== 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.5] 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDWvgFmO0NDGxAAsj0FxdzSOwqM3dys709CZg== 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.6] 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDZvgFmHWr/GBAABgIBP6Z5QiGlRkBFFFn/Rw== 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:[osd.7] 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDcvgFmxun5FhAAtlus2NtGjVDKv4Up+V6o6g== 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:[client.0] 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDDvgFmmWAPBBAA5UcdbVAv0J9z+7SAW0E0hA== 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout:[client.1] 2024-03-25T18:13:51.970 INFO:teuthology.orchestra.run.smithi007.stdout: key = AQDDvgFmzd5mBxAArKHVNVTk+7QyHhY46auCYQ== 2024-03-25T18:13:51.990 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.y] 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDCvgFm4VxAORAAq1hyBXq/wGAYljsMb5yT6w== 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.0] 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDEvgFmbf1lGhAAjLMj6gm8Zd78B5uR2XPC8Q== 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.1] 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDHvgFmw7ZAGBAAOQZ+wP5WMH+ioVXfpHaMPg== 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.2] 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDKvgFmtcquFhAAoPEGDzb+383//k5A9gaXxA== 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.3] 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDNvgFmUm4SFBAA0W/NwBGxcDyyI66ogpaAwg== 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout:[mgr.x] 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDDvgFm0wpgARAAASsumoiOoUSW/gKCuPcmeA== 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.4] 2024-03-25T18:13:51.991 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDTvgFm4u97HBAAGcwJA2m7Alfh6CcqBIgdwg== 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.5] 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDWvgFmO0NDGxAAsj0FxdzSOwqM3dys709CZg== 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.6] 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDZvgFmHWr/GBAABgIBP6Z5QiGlRkBFFFn/Rw== 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout:[osd.7] 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDcvgFmxun5FhAAtlus2NtGjVDKv4Up+V6o6g== 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout:[client.0] 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDDvgFmmWAPBBAA5UcdbVAv0J9z+7SAW0E0hA== 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout:[client.1] 2024-03-25T18:13:51.992 INFO:teuthology.orchestra.run.smithi183.stdout: key = AQDDvgFmzd5mBxAArKHVNVTk+7QyHhY46auCYQ== 2024-03-25T18:13:51.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:52.014 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:52.085 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.087 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.138 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.140 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.191 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.193 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.244 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.246 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.297 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:52.300 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-25T18:13:52.351 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.354 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.405 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.407 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.458 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.460 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.512 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.514 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-25T18:13:52.567 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:52.569 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:52.624 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:52.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-25T18:13:52.680 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-25T18:13:52.681 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-25T18:13:52.707 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:52.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-25T18:13:52.829 DEBUG:teuthology.orchestra.run.smithi007:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-25T18:13:52.896 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:52.993 DEBUG:teuthology.orchestra.run.smithi007:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-25T18:13:53.019 DEBUG:teuthology.orchestra.run.smithi183:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-25T18:13:53.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-25T18:13:53.143 DEBUG:teuthology.orchestra.run.smithi183:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-25T18:13:53.169 DEBUG:teuthology.orchestra.run.smithi007:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:53.172 DEBUG:teuthology.orchestra.run.smithi183:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-25T18:13:53.224 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-25T18:13:53.224 DEBUG:tasks.ceph_manager:running mon.a under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.a.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:53.224 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-25T18:13:53.224 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a 2024-03-25T18:13:53.226 INFO:tasks.ceph.mon.a:Started 2024-03-25T18:13:53.227 DEBUG:tasks.ceph_manager:running mon.c under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.c.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:53.227 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-25T18:13:53.227 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c 2024-03-25T18:13:53.229 INFO:tasks.ceph.mon.c:Started 2024-03-25T18:13:53.229 DEBUG:tasks.ceph_manager:running mon.b under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/mon.b.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck', '--leak-check=full', '--show-reachable=yes'] 2024-03-25T18:13:53.229 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-25T18:13:53.229 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b 2024-03-25T18:13:53.266 INFO:tasks.ceph.mon.b:Started 2024-03-25T18:13:53.266 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-25T18:13:53.266 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-25T18:13:53.266 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i y 2024-03-25T18:13:53.268 INFO:tasks.ceph.mgr.y:Started 2024-03-25T18:13:53.269 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-25T18:13:53.269 DEBUG:teuthology.orchestra.run.smithi183:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term ceph-mgr -f --cluster ceph -i x 2024-03-25T18:13:53.271 INFO:tasks.ceph.mgr.x:Started 2024-03-25T18:13:53.271 INFO:tasks.ceph:Setting crush tunables to default 2024-03-25T18:13:53.271 DEBUG:teuthology.orchestra.run.smithi007:> sudo ceph --cluster ceph osd crush tunables default 2024-03-25T18:13:53.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.371+0000 7f11f16ff640 10 monclient: get_monmap_and_config 2024-03-25T18:13:53.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.371+0000 7f11f16ff640 10 monclient: build_initial_monmap 2024-03-25T18:13:53.372 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.371+0000 7f11f16ff640 10 monclient: monmap: 2024-03-25T18:13:53.372 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:53.372433+0000 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:53.372433+0000 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 1 Processor -- start 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 1 -- start start 2024-03-25T18:13:53.373 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient: init 2024-03-25T18:13:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:13:53.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient(hunting): picked mon.noname-b con 0x7f11ec0b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient(hunting): picked mon.noname-a con 0x7f11ec13ff20 addr v1:172.21.15.7:6789/0 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient(hunting): picked mon.noname-c con 0x7f11ec142dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0ab440 con 0x7f11ec13ff20 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0ab6b0 con 0x7f11ec142dd0 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0a9f10 con 0x7f11ec0b4b50 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11f16ff640 10 monclient(hunting): authenticate will time out at 7275.477375s 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11ec13f810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11ec1426c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:53.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.372+0000 7f11eb7fe640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11ec14df50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:53.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.573+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11ec1426c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:53.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.573+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11ec13f810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:53.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.573+0000 7f11eb7fe640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11ec14df50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:53.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.974+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11ec13f810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:53.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.974+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11ec1426c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:53.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:53.974+0000 7f11eb7fe640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11ec14df50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:54.775+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11ec13f810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:54.775+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11ec1426c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:54.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:54.775+0000 7f11eb7fe640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11ec14df50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient(hunting): tick 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11ec13f810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11ec14df50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11ec1426c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f11ec142dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f11ec13ff20 addr v1:172.21.15.7:6789/0 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f11ec0b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:13:56.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0aa0a0 con 0x7f11ec13ff20 2024-03-25T18:13:56.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0ab440 con 0x7f11ec142dd0 2024-03-25T18:13:56.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0ab6b0 con 0x7f11ec0b4b50 2024-03-25T18:13:56.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11e8ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:56.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:56.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:56.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.372+0000 7f11eaffd640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:56.573 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.573+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:56.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.573+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:56.574 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.573+0000 7f11eaffd640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:56.974 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.974+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.974+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:56.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:56.974+0000 7f11eaffd640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:57.775 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:57.775+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d8008ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:57.775+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:57.776 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:57.775+0000 7f11eaffd640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.372+0000 7f11e8ff9640 10 monclient(hunting): tick 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.372+0000 7f11e8ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.372+0000 7f11e8ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.372+0000 7f11e8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.372+0000 7f11e8ff9640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.372+0000 7f11e8ff9640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.372+0000 7f11e8ff9640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d8008ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f11ec13ff20 addr v1:172.21.15.7:6789/0 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f11ec142dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f11ec0b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:13:59.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0a9f10 con 0x7f11ec13ff20 2024-03-25T18:13:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0aa0a0 con 0x7f11ec142dd0 2024-03-25T18:13:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0ab440 con 0x7f11ec0b4b50 2024-03-25T18:13:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11e8ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:13:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:59.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.373+0000 7f11ea7fc640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:59.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.574+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:59.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.574+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:59.575 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.574+0000 7f11ea7fc640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:13:59.975 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.975+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.975+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:13:59.976 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:13:59.975+0000 7f11ea7fc640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:14:00.198 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:00:06.825 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:00.502 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:00:07.129 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:00.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:00.776+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:14:00.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:00.776+0000 7f11eb7fe640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:14:00.777 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:00.776+0000 7f11ea7fc640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.183:6789/0 2024-03-25T18:14:00.855 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:07.493 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:00.856 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:07.493 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:01.183 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:00:07.820 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:01.183 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:00:07.820 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:14:02.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient(hunting): tick 2024-03-25T18:14:02.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:02.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:02.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d800c790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 1 -- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d80056d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f11ec0b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f11ec13ff20 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f11ec142dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0ab6b0 con 0x7f11ec13ff20 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0a9f10 con 0x7f11ec142dd0 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0aa0a0 con 0x7f11ec0b4b50 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11e8ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11eaffd640 1 -- >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d8009ea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6790/0 2024-03-25T18:14:02.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.373+0000 7f11ea7fc640 1 -- >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.7:6789/0 2024-03-25T18:14:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.428+0000 7f11eb7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 0x7f11d800c770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:43268/0 (socket says 172.21.15.7:43268) 2024-03-25T18:14:02.429 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.428+0000 7f11eb7fe640 1 -- 172.21.15.7:0/2787473422 learned_addr learned my addr 172.21.15.7:0/2787473422 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.986+0000 7f11eaffd640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d8009ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-25T18:14:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.986+0000 7f11eaffd640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d8009ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:14:02.987 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:02.986+0000 7f11eaffd640 1 --1- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 0x7f11d8009ea0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:14:03.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:03.063+0000 7f11ea7fc640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-25T18:14:03.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:03.063+0000 7f11ea7fc640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-25T18:14:03.063 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:03.063+0000 7f11ea7fc640 1 --1- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 0x7f11d800ffa0 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-25T18:14:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient(hunting): tick 2024-03-25T18:14:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:05.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6790/0 conn(0x7f11ec0b4b50 legacy=0x7f11d8009ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.183:6789/0 conn(0x7f11ec142dd0 legacy=0x7f11d800c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6789/0 conn(0x7f11ec13ff20 legacy=0x7f11d800ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f11d8005700 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f11d8012120 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f11d8009ea0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec0ab440 con 0x7f11d8009ea0 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d800a2d0 con 0x7f11d8005700 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8007520 con 0x7f11d8012120 2024-03-25T18:14:05.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:05.374+0000 7f11e8ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:08.374 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient(hunting): tick 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6790/0 conn(0x7f11d8012120 legacy=0x7f11d8014510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.183:6789/0 conn(0x7f11d8005700 legacy=0x7f11d8010810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6789/0 conn(0x7f11d8009ea0 legacy=0x7f11d8005e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f11d800c930 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f11d800df60 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f11d8017350 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d800c780 con 0x7f11d8017350 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8005e00 con 0x7f11d800df60 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8005fe0 con 0x7f11d800c930 2024-03-25T18:14:08.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:08.374+0000 7f11e8ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient(hunting): tick 2024-03-25T18:14:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:14:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 1 monclient(hunting): continuing hunt 2024-03-25T18:14:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:14:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6790/0 conn(0x7f11d800c930 legacy=0x7f11d80076b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.375 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.183:6789/0 conn(0x7f11d800df60 legacy=0x7f11d8015c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6789/0 conn(0x7f11d8017350 legacy=0x7f11d8019740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-c con 0x7f11d8010810 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-a con 0x7f11d8017350 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient(hunting): picked mon.noname-b con 0x7f11d800c930 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8006210 con 0x7f11d8017350 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8011f30 con 0x7f11d8010810 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11d8010c30 con 0x7f11d800c930 2024-03-25T18:14:11.376 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.374+0000 7f11e8ff9640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.441+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1435636246 0 0) 0x7f11d8006210 con 0x7f11d8017350 2024-03-25T18:14:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.441+0000 7f11e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.441+0000 7f11e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.441+0000 7f11e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:11.441 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.441+0000 7f11e9ffb640 10 monclient(hunting): my global_id is 4104 2024-03-25T18:14:11.442 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.441+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d4003520 con 0x7f11d8017350 2024-03-25T18:14:11.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.456+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4118169283 0 0) 0x7f11d8011f30 con 0x7f11d8010810 2024-03-25T18:14:11.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.456+0000 7f11e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:11.456 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.456+0000 7f11e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:11.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.456+0000 7f11e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:11.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.456+0000 7f11e9ffb640 10 monclient(hunting): my global_id is 4105 2024-03-25T18:14:11.457 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.456+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d8006210 con 0x7f11d8010810 2024-03-25T18:14:11.494 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.494+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3924299034 0 0) 0x7f11d8010c30 con 0x7f11d800c930 2024-03-25T18:14:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.494+0000 7f11e9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.494+0000 7f11e9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.494+0000 7f11e9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.494+0000 7f11e9ffb640 10 monclient(hunting): my global_id is 4112 2024-03-25T18:14:11.495 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.494+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d8011f30 con 0x7f11d800c930 2024-03-25T18:14:11.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2395854511 0 0) 0x7f11d8011f30 con 0x7f11d800c930 2024-03-25T18:14:11.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.183:6789/0 conn(0x7f11d8010810 legacy=0x7f11d8011110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.582 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6789/0 conn(0x7f11d8017350 legacy=0x7f11d8011820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 1 monclient: found mon.noname-b 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11d8019750 con 0x7f11d800c930 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11e4004490 con 0x7f11d800c930 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11e9ffb640 10 monclient: dump: 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:11.583 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11f16ff640 5 monclient: authenticate success, global_id 4112 2024-03-25T18:14:11.584 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.581+0000 7f11f16ff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11e4004660 con 0x7f11d800c930 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11e9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11e9ffb640 1 -- 172.21.15.7:0/2787473422 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11e4004a30 con 0x7f11d800c930 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11e9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11e9ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11e9ffb640 10 monclient: dump: 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:11.585 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:11.586 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:11.586 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:11.586 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:11.586 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:11.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11f16ff640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:11.586 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11f16ff640 4 set_mon_vals no callback set 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11f16ff640 10 monclient: shutdown 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11f16ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11f16ff640 1 -- 172.21.15.7:0/2787473422 >> v1:172.21.15.7:6790/0 conn(0x7f11d800c930 legacy=0x7f11d80076b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.584+0000 7f11f16ff640 1 -- 172.21.15.7:0/2787473422 shutdown_connections 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.585+0000 7f11f16ff640 1 -- 172.21.15.7:0/2787473422 >> 172.21.15.7:0/2787473422 conn(0x7f11ec066920 msgr2=0x7f11ec066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.585+0000 7f11f16ff640 1 -- 172.21.15.7:0/2787473422 shutdown_connections 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.585+0000 7f11f16ff640 1 -- 172.21.15.7:0/2787473422 wait complete. 2024-03-25T18:14:11.588 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.585+0000 7f11f16ff640 10 monclient: build_initial_monmap 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.585+0000 7f11f16ff640 10 monclient: monmap: 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.585+0000 7f11f16ff640 1 librados: starting msgr at 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.585+0000 7f11f16ff640 1 librados: starting objecter 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 1 Processor -- start 2024-03-25T18:14:11.589 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 1 -- start start 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 1 librados: setting wanted keys 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 1 librados: calling monclient init 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient: init 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient(hunting): picked mon.noname-a con 0x7f11d800c930 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient(hunting): picked mon.noname-b con 0x7f11d8010810 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient(hunting): picked mon.noname-c con 0x7f11d8017350 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec146170 con 0x7f11d800c930 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec145210 con 0x7f11d8010810 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11ec146600 con 0x7f11d8017350 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:11.590 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.586+0000 7f11f16ff640 10 monclient(hunting): authenticate will time out at 7293.691470s 2024-03-25T18:14:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.596+0000 7f11eb7fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f11d8017350 0x7f11ec1d8400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:44090/0 (socket says 172.21.15.7:44090) 2024-03-25T18:14:11.597 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.596+0000 7f11eb7fe640 1 -- 172.21.15.7:0/1351177772 learned_addr learned my addr 172.21.15.7:0/1351177772 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.612+0000 7f11d37fe640 10 client.?.objecter ms_handle_connect 0x7f11d8017350 2024-03-25T18:14:11.612 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.612+0000 7f11d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:11.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.616+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 177711370 0 0) 0x7f11ec146600 con 0x7f11d8017350 2024-03-25T18:14:11.616 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.616+0000 7f11d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.616+0000 7f11d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.616+0000 7f11d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.616+0000 7f11d37fe640 10 monclient(hunting): my global_id is 4121 2024-03-25T18:14:11.617 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.616+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11d8010e10 con 0x7f11d8017350 2024-03-25T18:14:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.618+0000 7f11d37fe640 10 client.?.objecter ms_handle_connect 0x7f11d800c930 2024-03-25T18:14:11.619 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.618+0000 7f11d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.632+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2744057889 0 0) 0x7f11d8010e10 con 0x7f11d8017350 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.632+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11ec146600 con 0x7f11d8017350 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.632+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11e4004970 con 0x7f11d8017350 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.632+0000 7f11d37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.632+0000 7f11d37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.632+0000 7f11d37fe640 10 monclient(hunting): dump: 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:11.633 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.632+0000 7f11d37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.633+0000 7f11d37fe640 10 client.?.objecter ms_handle_connect 0x7f11d8010810 2024-03-25T18:14:11.634 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.633+0000 7f11d37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.641+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2262380833 0 0) 0x7f11ec146600 con 0x7f11d8017350 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.641+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 >> v1:172.21.15.183:6789/0 conn(0x7f11d8010810 legacy=0x7f11ec1d7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.641+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 >> v1:172.21.15.7:6789/0 conn(0x7f11d800c930 legacy=0x7f11ec136f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 1 monclient: found mon.c 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11ec146790 con 0x7f11d8017350 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:11.643 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 5 monclient: authenticate success, global_id 4121 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 10 monclient: _renew_subs 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11ec13a320 con 0x7f11d8017350 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 10 client.4121.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 10 monclient: _renew_subs 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11ec137870 con 0x7f11d8017350 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.642+0000 7f11f16ff640 1 librados: init done 2024-03-25T18:14:11.644 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.643+0000 7f11f16ff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:11.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.643+0000 7f11f16ff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:11.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.643+0000 7f11f16ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:11.645 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.643+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11ec13a6c0 con 0x7f11d8017350 2024-03-25T18:14:11.901 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11e4004d80 con 0x7f11d8017350 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11e4005190 con 0x7f11d8017350 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 10 monclient: dump: 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:11.902 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f11e4005740 con 0x7f11d8017350 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3006060697 0 0) 0x7f11e40068a0 con 0x7f11d8017350 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 10 client.4121.objecter ms_dispatch 0x7f11ec000b90 osd_map(1..1 src has 1..1) v4 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 3 client.4121.objecter handle_osd_map got epochs [1,1] > 0 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 3 client.4121.objecter handle_osd_map decoding full epoch 1 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11e97fa640 4 set_mon_vals no callback set 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 20 client.4121.objecter dump_active .. 0 homeless 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f11e4018dc0 con 0x7f11d8017350 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:11.903 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.900+0000 7f11d37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:11.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.972+0000 7f11f16ff640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:11.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.972+0000 7f11f16ff640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:11.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.972+0000 7f11f16ff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:11.973 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:11.972+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f11ec13a6c0 con 0x7f11d8017350 2024-03-25T18:14:12.169 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:12.169+0000 7f3c176da040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:14:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.188+0000 7f11d37fe640 1 -- 172.21.15.7:0/1351177772 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7f11e4005d00 con 0x7f11d8017350 2024-03-25T18:14:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.188+0000 7f11d37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-25T18:14:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.188+0000 7f11d37fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-25T18:14:12.189 INFO:teuthology.orchestra.run.smithi007.stderr:adjusted tunables profile to default 2024-03-25T18:14:12.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 10 librados: watch_flush enter 2024-03-25T18:14:12.190 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 10 librados: watch_flush exit 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 20 client.4121.objecter shutdown clearing up homeless session... 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 10 client.4121.objecter successfully canceled tick 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 10 monclient: shutdown 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 >> v1:172.21.15.7:6790/0 conn(0x7f11d8017350 legacy=0x7f11ec1d8400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 shutdown_connections 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 >> 172.21.15.7:0/1351177772 conn(0x7f11ec066920 msgr2=0x7f11ec066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 shutdown_connections 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 1 -- 172.21.15.7:0/1351177772 wait complete. 2024-03-25T18:14:12.191 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:12.189+0000 7f11f16ff640 1 librados: shutdown 2024-03-25T18:14:12.201 INFO:tasks.ceph:check_enable_crimson: False 2024-03-25T18:14:12.201 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-25T18:14:12.201 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:14:12.201 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-25T18:14:12.228 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:14:12.228 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-25T18:14:12.230 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:12.230+0000 7f3c176da040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:14:12.258 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:14:12.258 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-25T18:14:12.328 DEBUG:teuthology.orchestra.run.smithi007:> set -ex 2024-03-25T18:14:12.328 DEBUG:teuthology.orchestra.run.smithi007:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-25T18:14:12.396 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:14:12.396 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-25T18:14:12.422 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:14:12.422 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-25T18:14:12.490 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:14:12.490 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-25T18:14:12.558 DEBUG:teuthology.orchestra.run.smithi183:> set -ex 2024-03-25T18:14:12.558 DEBUG:teuthology.orchestra.run.smithi183:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-25T18:14:12.601 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:12.600+0000 7ff6ff045040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-25T18:14:12.626 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 03af994c-009d-49db-af60-10fa9c6fc71e 0 2024-03-25T18:14:12.659 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:12.658+0000 7ff6ff045040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-25T18:14:12.760 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.759+0000 7f485fd2b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:12.760 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.759+0000 7f485fd2b640 10 monclient: build_initial_monmap 2024-03-25T18:14:12.760 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.759+0000 7f485fd2b640 10 monclient: monmap: 2024-03-25T18:14:12.760 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:12.760 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:12.760915+0000 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:12.760915+0000 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:12.761 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:12.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.760+0000 7f485fd2b640 1 Processor -- start 2024-03-25T18:14:12.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.760+0000 7f485fd2b640 1 -- start start 2024-03-25T18:14:12.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.760+0000 7f485fd2b640 10 monclient: init 2024-03-25T18:14:12.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 10 monclient(hunting): picked mon.noname-a con 0x7f48580b4f00 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 10 monclient(hunting): picked mon.noname-c con 0x7f48580b4510 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 10 monclient(hunting): picked mon.noname-b con 0x7f4858142ad0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580a9d90 con 0x7f48580b4f00 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48580aa2b0 con 0x7f48580b4510 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f485813fc60 con 0x7f4858142ad0 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485fd2b640 10 monclient(hunting): authenticate will time out at 7297.035103s 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485e2c8640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f4858142ad0 0x7f485814dc80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.183:37612/0 (socket says 172.21.15.183:37612) 2024-03-25T18:14:12.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.761+0000 7f485e2c8640 1 -- 172.21.15.183:0/1324666537 learned_addr learned my addr 172.21.15.183:0/1324666537 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2698047595 0 0) 0x7f485813fc60 con 0x7f4858142ad0 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): my global_id is 4124 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4844003520 con 0x7f4858142ad0 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1851815577 0 0) 0x7f48580a9d90 con 0x7f48580b4f00 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 10 monclient(hunting): my global_id is 4110 2024-03-25T18:14:12.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.764+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f485813fc60 con 0x7f48580b4f00 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 625227659 0 0) 0x7f4844003520 con 0x7f4858142ad0 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 >> v1:172.21.15.183:6789/0 conn(0x7f48580b4510 legacy=0x7f485813f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 >> v1:172.21.15.7:6789/0 conn(0x7f48580b4f00 legacy=0x7f48581423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 10 monclient: _finish_hunting 0 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 1 monclient: found mon.noname-b 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f485813fdf0 con 0x7f4858142ad0 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 10 monclient: _finish_auth 0 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 10 monclient: _check_auth_tickets 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4850001a10 con 0x7f4858142ad0 2024-03-25T18:14:12.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485fd2b640 5 monclient: authenticate success, global_id 4124 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485fd2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485cac5640 10 monclient: dump: 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:12.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.767+0000 7f485fd2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4850001b70 con 0x7f4858142ad0 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485cac5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485cac5640 1 -- 172.21.15.183:0/1324666537 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4850001f80 con 0x7f4858142ad0 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485cac5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485cac5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485cac5640 10 monclient: dump: 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.774 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:12.775 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:12.775 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:12.775 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:12.775 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:12.775 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485fd2b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:12.775 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.773+0000 7f485fd2b640 4 set_mon_vals no callback set 2024-03-25T18:14:12.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 10 monclient: shutdown 2024-03-25T18:14:12.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:12.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 1 -- 172.21.15.183:0/1324666537 >> v1:172.21.15.7:6790/0 conn(0x7f4858142ad0 legacy=0x7f485814dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 1 -- 172.21.15.183:0/1324666537 shutdown_connections 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 1 -- 172.21.15.183:0/1324666537 >> 172.21.15.183:0/1324666537 conn(0x7f4858066b00 msgr2=0x7f4858066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 1 -- 172.21.15.183:0/1324666537 shutdown_connections 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 1 -- 172.21.15.183:0/1324666537 wait complete. 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 10 monclient: build_initial_monmap 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.774+0000 7f485fd2b640 10 monclient: monmap: 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:12.778 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 librados: starting msgr at 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 librados: starting objecter 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 Processor -- start 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 -- start start 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 librados: setting wanted keys 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 librados: calling monclient init 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient: init 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient(hunting): picked mon.noname-b con 0x7f48580b4510 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient(hunting): picked mon.noname-a con 0x7f48580b4f00 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient(hunting): picked mon.noname-c con 0x7f4858142ad0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48581d4e10 con 0x7f48580b4f00 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48581455a0 con 0x7f48580b4510 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4858145cf0 con 0x7f4858142ad0 2024-03-25T18:14:12.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.775+0000 7f485fd2b640 10 monclient(hunting): authenticate will time out at 7297.049683s 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f485dac7640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f48580b4510 0x7f4858143aa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38568/0 (socket says 172.21.15.183:38568) 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f485dac7640 1 -- 172.21.15.183:0/1578955782 learned_addr learned my addr 172.21.15.183:0/1578955782 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f484e7fc640 10 client.?.objecter ms_handle_connect 0x7f48580b4510 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f484e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f484e7fc640 10 client.?.objecter ms_handle_connect 0x7f48580b4f00 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f484e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f484e7fc640 10 client.?.objecter ms_handle_connect 0x7f4858142ad0 2024-03-25T18:14:12.780 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.776+0000 7f484e7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.778+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891828610 0 0) 0x7f48581455a0 con 0x7f48580b4510 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.778+0000 7f484e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.778+0000 7f484e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.778+0000 7f484e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.778+0000 7f484e7fc640 10 monclient(hunting): my global_id is 4111 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.778+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f483c004eb0 con 0x7f48580b4510 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3715389130 0 0) 0x7f48581d4e10 con 0x7f48580b4f00 2024-03-25T18:14:12.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): my global_id is 4113 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48581455a0 con 0x7f48580b4f00 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 264813820 0 0) 0x7f4858145cf0 con 0x7f4858142ad0 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 10 monclient(hunting): my global_id is 4127 2024-03-25T18:14:12.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.779+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48581d4e10 con 0x7f4858142ad0 2024-03-25T18:14:12.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.783+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3619680601 0 0) 0x7f483c004eb0 con 0x7f48580b4510 2024-03-25T18:14:12.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.783+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4858145cf0 con 0x7f48580b4510 2024-03-25T18:14:12.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.783+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4854001a10 con 0x7f48580b4510 2024-03-25T18:14:12.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.783+0000 7f484e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.783+0000 7f484e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:12.784 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.783+0000 7f484e7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:12.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.783+0000 7f484e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:12.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.785+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1860601488 0 0) 0x7f48581455a0 con 0x7f48580b4f00 2024-03-25T18:14:12.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.785+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f483c004eb0 con 0x7f48580b4f00 2024-03-25T18:14:12.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.785+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4848001810 con 0x7f48580b4f00 2024-03-25T18:14:12.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.785+0000 7f484e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.785+0000 7f484e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:12.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.785+0000 7f484e7fc640 10 monclient(hunting): dump: 2024-03-25T18:14:12.786 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:12.787 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 100105155 0 0) 0x7f4858145cf0 con 0x7f48580b4510 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 >> v1:172.21.15.7:6790/0 conn(0x7f4858142ad0 legacy=0x7f48581d4700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 >> v1:172.21.15.7:6789/0 conn(0x7f48580b4f00 legacy=0x7f48581d7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 1 monclient: found mon.b 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:12.788 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48581459c0 con 0x7f48580b4510 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f484e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f485fd2b640 5 monclient: authenticate success, global_id 4111 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.786+0000 7f485fd2b640 10 monclient: _renew_subs 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.787+0000 7f485fd2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.787+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48581442b0 con 0x7f48580b4510 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.787+0000 7f485fd2b640 10 client.4111.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.787+0000 7f485fd2b640 10 monclient: _renew_subs 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.787+0000 7f485fd2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.787+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4858143690 con 0x7f48580b4510 2024-03-25T18:14:12.789 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.787+0000 7f485fd2b640 1 librados: init done 2024-03-25T18:14:12.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.788+0000 7f485fd2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.788+0000 7f485fd2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.788+0000 7f485fd2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:12.790 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.788+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4858145e80 con 0x7f48580b4510 2024-03-25T18:14:12.838 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:12.838+0000 7f3c176da040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:14:12.895 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:12.894+0000 7f3c176da040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:14:12.942 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:12.941+0000 7f3c176da040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:14:12.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4854001c50 con 0x7f48580b4510 2024-03-25T18:14:12.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4854003a60 con 0x7f48580b4510 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 10 monclient: dump: 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:12.964 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4854004010 con 0x7f48580b4510 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484ffff640 4 set_mon_vals no callback set 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3532168975 0 0) 0x7f48540065b0 con 0x7f48580b4510 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 10 client.4111.objecter ms_dispatch 0x7f4858000b90 osd_map(2..2 src has 1..2) v4 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 3 client.4111.objecter handle_osd_map got epochs [2,2] > 0 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 3 client.4111.objecter handle_osd_map decoding full epoch 2 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 20 client.4111.objecter dump_active .. 0 homeless 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f48540069b0 con 0x7f48580b4510 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:12.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:12.962+0000 7f484e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:12.997 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:12.997+0000 7f3c176da040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:14:13.036 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.035+0000 7f485fd2b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "03af994c-009d-49db-af60-10fa9c6fc71e", "id": 0}] 2024-03-25T18:14:13.036 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.035+0000 7f485fd2b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "03af994c-009d-49db-af60-10fa9c6fc71e", "id": 0}] 2024-03-25T18:14:13.036 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.035+0000 7f485fd2b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:13.036 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.035+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "03af994c-009d-49db-af60-10fa9c6fc71e", "id": 0} v 0) v1 -- 0x7f485804f730 con 0x7f48580b4510 2024-03-25T18:14:13.041 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.041+0000 7f3c176da040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:14:13.163 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.162+0000 7f3c176da040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:14:13.203 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.202+0000 7ff6ff045040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-25T18:14:13.237 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.237+0000 7f3c176da040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:13.239 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.238+0000 7f484e7fc640 1 -- 172.21.15.183:0/1578955782 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "03af994c-009d-49db-af60-10fa9c6fc71e", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 3265655943 0 732707515) 0x7f4854006b50 con 0x7f48580b4510 2024-03-25T18:14:13.239 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.238+0000 7f484e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "03af994c-009d-49db-af60-10fa9c6fc71e", "id": 0}] 2024-03-25T18:14:13.239 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.238+0000 7f484e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:13.239 INFO:teuthology.orchestra.run.smithi183.stdout:0 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 10 librados: watch_flush enter 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 10 librados: watch_flush exit 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 20 client.4111.objecter shutdown clearing up homeless session... 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 10 client.4111.objecter successfully canceled tick 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 10 monclient: shutdown 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 >> v1:172.21.15.183:6789/0 conn(0x7f48580b4510 legacy=0x7f4858143aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 shutdown_connections 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 >> 172.21.15.183:0/1578955782 conn(0x7f4858066b00 msgr2=0x7f48581402c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 shutdown_connections 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 1 -- 172.21.15.183:0/1578955782 wait complete. 2024-03-25T18:14:13.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.239+0000 7f485fd2b640 1 librados: shutdown 2024-03-25T18:14:13.251 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new f245fce3-4660-44a7-8c53-39a3a2091c33 1 2024-03-25T18:14:13.261 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.260+0000 7ff6ff045040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-25T18:14:13.285 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.284+0000 7f3c176da040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:14:13.308 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.308+0000 7ff6ff045040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-25T18:14:13.339 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.338+0000 7f3c176da040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:14:13.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.349+0000 7f9ff2fe8640 10 monclient: get_monmap_and_config 2024-03-25T18:14:13.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.349+0000 7f9ff2fe8640 10 monclient: build_initial_monmap 2024-03-25T18:14:13.350 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.349+0000 7f9ff2fe8640 10 monclient: monmap: 2024-03-25T18:14:13.350 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:13.350 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.350 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:13.350653+0000 2024-03-25T18:14:13.350 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:13.350653+0000 2024-03-25T18:14:13.351 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:13.351 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.351 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:13.351 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:13.351 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:13.351 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 1 Processor -- start 2024-03-25T18:14:13.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 1 -- start start 2024-03-25T18:14:13.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient: init 2024-03-25T18:14:13.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:13.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:13.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec0b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:13.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec13ff20 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0ab440 con 0x7f9fec142dd0 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0ab6b0 con 0x7f9fec13ff20 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec0a9f10 con 0x7f9fec0b4b50 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.350+0000 7f9ff2fe8640 10 monclient(hunting): authenticate will time out at 7297.624770s 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.351+0000 7f9febfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f9fec13ff20 0x7f9fec1426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38582/0 (socket says 172.21.15.183:38582) 2024-03-25T18:14:13.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.351+0000 7f9febfff640 1 -- 172.21.15.183:0/2041302900 learned_addr learned my addr 172.21.15.183:0/2041302900 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1616194373 0 0) 0x7f9fec0ab440 con 0x7f9fec142dd0 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): my global_id is 4116 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fdc003520 con 0x7f9fec142dd0 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 165223341 0 0) 0x7f9fec0ab6b0 con 0x7f9fec13ff20 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): my global_id is 4114 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec0ab440 con 0x7f9fec13ff20 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1337884321 0 0) 0x7f9fec0a9f10 con 0x7f9fec0b4b50 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 10 monclient(hunting): my global_id is 4130 2024-03-25T18:14:13.356 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.354+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec0ab6b0 con 0x7f9fec0b4b50 2024-03-25T18:14:13.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 531247568 0 0) 0x7f9fdc003520 con 0x7f9fec142dd0 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 >> v1:172.21.15.7:6790/0 conn(0x7f9fec0b4b50 legacy=0x7f9fec13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 >> v1:172.21.15.183:6789/0 conn(0x7f9fec13ff20 legacy=0x7f9fec1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 1 monclient: found mon.noname-a 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec0aa0a0 con 0x7f9fec142dd0 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9fe4001a10 con 0x7f9fec142dd0 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9feb7fe640 10 monclient: dump: 2024-03-25T18:14:13.359 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9ff2fe8640 5 monclient: authenticate success, global_id 4116 2024-03-25T18:14:13.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.357+0000 7f9ff2fe8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:13.367 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.366+0000 7ff6ff045040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-25T18:14:13.385 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe4001b70 con 0x7f9fec142dd0 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9feb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9feb7fe640 1 -- 172.21.15.183:0/2041302900 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9fe4001f00 con 0x7f9fec142dd0 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9ff2fe8640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9ff2fe8640 4 set_mon_vals no callback set 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9feb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9feb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.384+0000 7f9feb7fe640 10 monclient: dump: 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.386 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 10 monclient: shutdown 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/2041302900 >> v1:172.21.15.7:6789/0 conn(0x7f9fec142dd0 legacy=0x7f9fec14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/2041302900 shutdown_connections 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/2041302900 >> 172.21.15.183:0/2041302900 conn(0x7f9fec066920 msgr2=0x7f9fec066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/2041302900 shutdown_connections 2024-03-25T18:14:13.387 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/2041302900 wait complete. 2024-03-25T18:14:13.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 10 monclient: build_initial_monmap 2024-03-25T18:14:13.389 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.385+0000 7f9ff2fe8640 10 monclient: monmap: 2024-03-25T18:14:13.389 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:13.389 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:13.389 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:13.389 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 librados: starting msgr at 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 librados: starting objecter 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 Processor -- start 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 -- start start 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 librados: setting wanted keys 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 librados: calling monclient init 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient: init 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:13.390 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient(hunting): picked mon.noname-a con 0x7f9fec0b4b50 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient(hunting): picked mon.noname-b con 0x7f9fec13ff20 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient(hunting): picked mon.noname-c con 0x7f9fec142dd0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec13a900 con 0x7f9fec0b4b50 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec134c40 con 0x7f9fec13ff20 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fec144a30 con 0x7f9fec142dd0 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.386+0000 7f9ff2fe8640 10 monclient(hunting): authenticate will time out at 7297.660713s 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.387+0000 7f9febfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f9fec13ff20 0x7f9fec133e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38594/0 (socket says 172.21.15.183:38594) 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.387+0000 7f9febfff640 1 -- 172.21.15.183:0/349864688 learned_addr learned my addr 172.21.15.183:0/349864688 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.387+0000 7f9fe97fa640 10 client.?.objecter ms_handle_connect 0x7f9fec13ff20 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.387+0000 7f9fe97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.387+0000 7f9fe97fa640 10 client.?.objecter ms_handle_connect 0x7f9fec0b4b50 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.387+0000 7f9fe97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.388+0000 7f9fe97fa640 10 client.?.objecter ms_handle_connect 0x7f9fec142dd0 2024-03-25T18:14:13.391 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.388+0000 7f9fe97fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:13.392 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.390+0000 7f3c176da040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.389+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2244076720 0 0) 0x7f9fec13a900 con 0x7f9fec0b4b50 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.389+0000 7f9fe97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.389+0000 7f9fe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.389+0000 7f9fe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.389+0000 7f9fe97fa640 10 monclient(hunting): my global_id is 4119 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.389+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fd0004eb0 con 0x7f9fec0b4b50 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2870951473 0 0) 0x7f9fec134c40 con 0x7f9fec13ff20 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): my global_id is 4117 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec13a900 con 0x7f9fec13ff20 2024-03-25T18:14:13.393 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3525010863 0 0) 0x7f9fec144a30 con 0x7f9fec142dd0 2024-03-25T18:14:13.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 10 monclient(hunting): my global_id is 4133 2024-03-25T18:14:13.394 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.390+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9fec134c40 con 0x7f9fec142dd0 2024-03-25T18:14:13.396 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007149184 0 0) 0x7f9fd0004eb0 con 0x7f9fec0b4b50 2024-03-25T18:14:13.396 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fec144a30 con 0x7f9fec0b4b50 2024-03-25T18:14:13.396 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9fe00018f0 con 0x7f9fec0b4b50 2024-03-25T18:14:13.396 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.396 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): dump: 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2121906006 0 0) 0x7f9fec134c40 con 0x7f9fec142dd0 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fd0004eb0 con 0x7f9fec142dd0 2024-03-25T18:14:13.397 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9fe4001ce0 con 0x7f9fec142dd0 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): dump: 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.398 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3399497015 0 0) 0x7f9fec13a900 con 0x7f9fec13ff20 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9fec134c40 con 0x7f9fec13ff20 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9fd8001970 con 0x7f9fec13ff20 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.395+0000 7f9fe97fa640 10 monclient(hunting): dump: 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.399 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.400 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2418911170 0 0) 0x7f9fec144a30 con 0x7f9fec0b4b50 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 >> v1:172.21.15.7:6790/0 conn(0x7f9fec142dd0 legacy=0x7f9fec134530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 >> v1:172.21.15.183:6789/0 conn(0x7f9fec13ff20 legacy=0x7f9fec133e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 1 monclient: found mon.a 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fec144d10 con 0x7f9fec0b4b50 2024-03-25T18:14:13.401 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9fe97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9ff2fe8640 5 monclient: authenticate success, global_id 4119 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9ff2fe8640 10 monclient: _renew_subs 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.397+0000 7f9ff2fe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fec13ad90 con 0x7f9fec0b4b50 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 10 client.4119.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 10 monclient: _renew_subs 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fec139d50 con 0x7f9fec0b4b50 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 1 librados: init done 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:13.402 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.398+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fec04f730 con 0x7f9fec0b4b50 2024-03-25T18:14:13.413 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.412+0000 7ff6ff045040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-25T18:14:13.529 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.528+0000 7ff6ff045040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9fe0003610 con 0x7f9fec0b4b50 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9fe97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9fe0001d80 con 0x7f9fec0b4b50 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9fe97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9fe97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9fe97fa640 10 monclient: dump: 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.585 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9fe0003d70 con 0x7f9fec0b4b50 2024-03-25T18:14:13.586 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.584+0000 7f9feaffd640 4 set_mon_vals no callback set 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 689712246 0 0) 0x7f9fe00053a0 con 0x7f9fec0b4b50 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 10 client.4119.objecter ms_dispatch 0x7f9fec000b90 osd_map(3..3 src has 1..3) v4 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 3 client.4119.objecter handle_osd_map got epochs [3,3] > 0 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 3 client.4119.objecter handle_osd_map decoding full epoch 3 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 20 client.4119.objecter dump_active .. 0 homeless 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9fe00056f0 con 0x7f9fec0b4b50 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:13.589 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.588+0000 7f9fe97fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:13.600 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.599+0000 7ff6ff045040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:13.648 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.648+0000 7ff6ff045040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-25T18:14:13.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.660+0000 7f9ff2fe8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "f245fce3-4660-44a7-8c53-39a3a2091c33", "id": 1}] 2024-03-25T18:14:13.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.660+0000 7f9ff2fe8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "f245fce3-4660-44a7-8c53-39a3a2091c33", "id": 1}] 2024-03-25T18:14:13.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.660+0000 7f9ff2fe8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:13.661 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.660+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd new", "uuid": "f245fce3-4660-44a7-8c53-39a3a2091c33", "id": 1} v 0) v1 -- 0x7f9fec0b6460 con 0x7f9fec0b4b50 2024-03-25T18:14:13.701 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.700+0000 7ff6ff045040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-25T18:14:13.748 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:13.748+0000 7ff6ff045040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-25T18:14:13.797 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.796+0000 7f3c176da040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:14:13.803 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.802+0000 7f9fe97fa640 1 -- 172.21.15.183:0/349864688 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "f245fce3-4660-44a7-8c53-39a3a2091c33", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 3078636481 0 940483276) 0x7f9fe0005930 con 0x7f9fec0b4b50 2024-03-25T18:14:13.803 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.802+0000 7f9fe97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "f245fce3-4660-44a7-8c53-39a3a2091c33", "id": 1}] 2024-03-25T18:14:13.803 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.802+0000 7f9fe97fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:13.804 INFO:teuthology.orchestra.run.smithi183.stdout:1 2024-03-25T18:14:13.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 10 librados: watch_flush enter 2024-03-25T18:14:13.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 10 librados: watch_flush exit 2024-03-25T18:14:13.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 20 client.4119.objecter shutdown clearing up homeless session... 2024-03-25T18:14:13.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 10 client.4119.objecter successfully canceled tick 2024-03-25T18:14:13.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 10 monclient: shutdown 2024-03-25T18:14:13.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:13.806 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 >> v1:172.21.15.7:6789/0 conn(0x7f9fec0b4b50 legacy=0x7f9fec1db0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.806 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 shutdown_connections 2024-03-25T18:14:13.806 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 >> 172.21.15.183:0/349864688 conn(0x7f9fec066920 msgr2=0x7f9fec0b34a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:13.806 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 shutdown_connections 2024-03-25T18:14:13.806 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.803+0000 7f9ff2fe8640 1 -- 172.21.15.183:0/349864688 wait complete. 2024-03-25T18:14:13.806 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.804+0000 7f9ff2fe8640 1 librados: shutdown 2024-03-25T18:14:13.815 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 1b46f77c-50d6-4e5b-8c25-a5ae755599fc 2 2024-03-25T18:14:13.943 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:13.942+0000 7f3c176da040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:14:13.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.955+0000 7f2ca8b4b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.955+0000 7f2ca8b4b640 10 monclient: build_initial_monmap 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.955+0000 7f2ca8b4b640 10 monclient: monmap: 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:13.956838+0000 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:13.956838+0000 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:13.957 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.956+0000 7f2ca8b4b640 1 Processor -- start 2024-03-25T18:14:13.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.956+0000 7f2ca8b4b640 1 -- start start 2024-03-25T18:14:13.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.956+0000 7f2ca8b4b640 10 monclient: init 2024-03-25T18:14:13.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:13.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:13.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ca40b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f2ca413ff20 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f2ca4142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40ab440 con 0x7f2ca4142dd0 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40ab6b0 con 0x7f2ca413ff20 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40a9f10 con 0x7f2ca40b4b50 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2ca8b4b640 10 monclient(hunting): authenticate will time out at 7298.231043s 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2c9bfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2ca413ff20 0x7f2ca41426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38598/0 (socket says 172.21.15.183:38598) 2024-03-25T18:14:13.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.957+0000 7f2c9bfff640 1 -- 172.21.15.183:0/3296332980 learned_addr learned my addr 172.21.15.183:0/3296332980 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:13.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2055684006 0 0) 0x7f2ca40ab6b0 con 0x7f2ca413ff20 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): my global_id is 4120 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c90003520 con 0x7f2ca413ff20 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190529113 0 0) 0x7f2ca40ab440 con 0x7f2ca4142dd0 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): my global_id is 4122 2024-03-25T18:14:13.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca40ab6b0 con 0x7f2ca4142dd0 2024-03-25T18:14:13.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3374525260 0 0) 0x7f2ca40a9f10 con 0x7f2ca40b4b50 2024-03-25T18:14:13.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 10 monclient(hunting): my global_id is 4136 2024-03-25T18:14:13.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.960+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca40ab440 con 0x7f2ca40b4b50 2024-03-25T18:14:13.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2258334904 0 0) 0x7f2ca40ab440 con 0x7f2ca40b4b50 2024-03-25T18:14:13.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 >> v1:172.21.15.183:6789/0 conn(0x7f2ca413ff20 legacy=0x7f2ca41426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 >> v1:172.21.15.7:6789/0 conn(0x7f2ca4142dd0 legacy=0x7f2ca414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 1 monclient: found mon.noname-b 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40aa0a0 con 0x7f2ca40b4b50 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 10 monclient: _finish_auth 0 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2c940015b0 con 0x7f2ca40b4b50 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca1d9b640 10 monclient: dump: 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca8b4b640 5 monclient: authenticate success, global_id 4136 2024-03-25T18:14:13.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.963+0000 7f2ca8b4b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:13.967 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c94001710 con 0x7f2ca40b4b50 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca1d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca1d9b640 1 -- 172.21.15.183:0/3296332980 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2c94001b60 con 0x7f2ca40b4b50 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 4 set_mon_vals no callback set 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca1d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca1d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca1d9b640 10 monclient: dump: 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.968 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 10 monclient: shutdown 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/3296332980 >> v1:172.21.15.7:6790/0 conn(0x7f2ca40b4b50 legacy=0x7f2ca413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/3296332980 shutdown_connections 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/3296332980 >> 172.21.15.183:0/3296332980 conn(0x7f2ca4066920 msgr2=0x7f2ca4066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/3296332980 shutdown_connections 2024-03-25T18:14:13.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.966+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/3296332980 wait complete. 2024-03-25T18:14:13.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.967+0000 7f2ca8b4b640 10 monclient: build_initial_monmap 2024-03-25T18:14:13.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.967+0000 7f2ca8b4b640 10 monclient: monmap: 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.967+0000 7f2ca8b4b640 1 librados: starting msgr at 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.967+0000 7f2ca8b4b640 1 librados: starting objecter 2024-03-25T18:14:13.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 1 Processor -- start 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 1 -- start start 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 1 librados: setting wanted keys 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 1 librados: calling monclient init 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient: init 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient(hunting): picked mon.noname-b con 0x7f2ca40b4b50 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient(hunting): picked mon.noname-c con 0x7f2ca413ff20 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient(hunting): picked mon.noname-a con 0x7f2ca4142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca4146500 con 0x7f2ca4142dd0 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca4140ac0 con 0x7f2ca40b4b50 2024-03-25T18:14:13.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ca40b16d0 con 0x7f2ca413ff20 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.968+0000 7f2ca8b4b640 10 monclient(hunting): authenticate will time out at 7298.242409s 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2ca259c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f2ca40b4b50 0x7f2ca4144380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38610/0 (socket says 172.21.15.183:38610) 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2ca259c640 1 -- 172.21.15.183:0/221954193 learned_addr learned my addr 172.21.15.183:0/221954193 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2c9affd640 10 client.?.objecter ms_handle_connect 0x7f2ca40b4b50 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2c9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2c9affd640 10 client.?.objecter ms_handle_connect 0x7f2ca4142dd0 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2c9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2c9affd640 10 client.?.objecter ms_handle_connect 0x7f2ca413ff20 2024-03-25T18:14:13.974 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.969+0000 7f2c9affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:13.975 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1912413520 0 0) 0x7f2ca4140ac0 con 0x7f2ca40b4b50 2024-03-25T18:14:13.975 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.975 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): my global_id is 4123 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c84004eb0 con 0x7f2ca40b4b50 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3106934235 0 0) 0x7f2ca40b16d0 con 0x7f2ca413ff20 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 10 monclient(hunting): my global_id is 4139 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.971+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca4140ac0 con 0x7f2ca413ff20 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.972+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556547101 0 0) 0x7f2ca4146500 con 0x7f2ca4142dd0 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.972+0000 7f2c9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.972+0000 7f2c9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:13.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.972+0000 7f2c9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:13.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.972+0000 7f2c9affd640 10 monclient(hunting): my global_id is 4125 2024-03-25T18:14:13.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.972+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ca40b16d0 con 0x7f2ca4142dd0 2024-03-25T18:14:13.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.977+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 215082207 0 0) 0x7f2c84004eb0 con 0x7f2ca40b4b50 2024-03-25T18:14:13.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.977+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ca4146500 con 0x7f2ca40b4b50 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.977+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2c94001b20 con 0x7f2ca40b4b50 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.977+0000 7f2c9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.977+0000 7f2c9affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.977+0000 7f2c9affd640 10 monclient(hunting): dump: 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.978 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.979 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.979 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.979 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.977+0000 7f2c9affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1958640554 0 0) 0x7f2ca4140ac0 con 0x7f2ca413ff20 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2c84004eb0 con 0x7f2ca413ff20 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2c8c001950 con 0x7f2ca413ff20 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 10 monclient(hunting): dump: 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.980 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296484419 0 0) 0x7f2ca40b16d0 con 0x7f2ca4142dd0 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ca4140ac0 con 0x7f2ca4142dd0 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2c9c001bd0 con 0x7f2ca4142dd0 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.979+0000 7f2c9affd640 10 monclient(hunting): dump: 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:13.981 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:13.982 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:13.982 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:13.982 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:13.982 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:13.982 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:13.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3533609595 0 0) 0x7f2ca4146500 con 0x7f2ca40b4b50 2024-03-25T18:14:13.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 >> v1:172.21.15.7:6790/0 conn(0x7f2ca413ff20 legacy=0x7f2ca40b0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 >> v1:172.21.15.7:6789/0 conn(0x7f2ca4142dd0 legacy=0x7f2ca4140fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:13.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:13.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 1 monclient: found mon.b 2024-03-25T18:14:13.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:13.983 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ca40b1860 con 0x7f2ca40b4b50 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2c9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 5 monclient: authenticate success, global_id 4123 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 10 monclient: _renew_subs 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ca4145280 con 0x7f2ca40b4b50 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 10 client.4123.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 10 monclient: _renew_subs 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ca41416f0 con 0x7f2ca40b4b50 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.981+0000 7f2ca8b4b640 1 librados: init done 2024-03-25T18:14:13.984 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.982+0000 7f2ca8b4b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:13.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.982+0000 7f2ca8b4b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:13.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.982+0000 7f2ca8b4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:13.985 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:13.982+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ca4140c50 con 0x7f2ca40b4b50 2024-03-25T18:14:14.060 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:14.059+0000 7f3c176da040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:14:14.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2c940038a0 con 0x7f2ca40b4b50 2024-03-25T18:14:14.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:14.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2c94003c30 con 0x7f2ca40b4b50 2024-03-25T18:14:14.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:14.081 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 10 monclient: dump: 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2c940041e0 con 0x7f2ca40b4b50 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2ca159a640 4 set_mon_vals no callback set 2024-03-25T18:14:14.082 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 1261293656 0 0) 0x7f2c94006730 con 0x7f2ca40b4b50 2024-03-25T18:14:14.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 10 client.4123.objecter ms_dispatch 0x7f2ca4000b90 osd_map(4..4 src has 1..4) v4 2024-03-25T18:14:14.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 3 client.4123.objecter handle_osd_map got epochs [4,4] > 0 2024-03-25T18:14:14.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 3 client.4123.objecter handle_osd_map decoding full epoch 4 2024-03-25T18:14:14.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 20 client.4123.objecter dump_active .. 0 homeless 2024-03-25T18:14:14.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2c94006b30 con 0x7f2ca40b4b50 2024-03-25T18:14:14.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.083 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.080+0000 7f2c9affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:14.106 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:14.106+0000 7f3c176da040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:14:14.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.153+0000 7f2ca8b4b640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "1b46f77c-50d6-4e5b-8c25-a5ae755599fc", "id": 2}] 2024-03-25T18:14:14.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.153+0000 7f2ca8b4b640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "1b46f77c-50d6-4e5b-8c25-a5ae755599fc", "id": 2}] 2024-03-25T18:14:14.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.153+0000 7f2ca8b4b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:14.154 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.153+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "1b46f77c-50d6-4e5b-8c25-a5ae755599fc", "id": 2} v 0) v1 -- 0x7f2ca404f730 con 0x7f2ca40b4b50 2024-03-25T18:14:14.155 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.154+0000 7ff6ff045040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-25T18:14:14.227 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:14.227+0000 7f3c176da040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:14.240 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.240+0000 7f2c9affd640 1 -- 172.21.15.183:0/221954193 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1b46f77c-50d6-4e5b-8c25-a5ae755599fc", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 3111369013 0 216599125) 0x7f2c94006310 con 0x7f2ca40b4b50 2024-03-25T18:14:14.240 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.240+0000 7f2c9affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "1b46f77c-50d6-4e5b-8c25-a5ae755599fc", "id": 2}] 2024-03-25T18:14:14.241 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.240+0000 7f2c9affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:14.241 INFO:teuthology.orchestra.run.smithi183.stdout:2 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 10 librados: watch_flush enter 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 10 librados: watch_flush exit 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 20 client.4123.objecter shutdown clearing up homeless session... 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 10 client.4123.objecter successfully canceled tick 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 10 monclient: shutdown 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 >> v1:172.21.15.183:6789/0 conn(0x7f2ca40b4b50 legacy=0x7f2ca4144380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 shutdown_connections 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 >> 172.21.15.183:0/221954193 conn(0x7f2ca4066920 msgr2=0x7f2ca4066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 shutdown_connections 2024-03-25T18:14:14.242 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 1 -- 172.21.15.183:0/221954193 wait complete. 2024-03-25T18:14:14.243 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.241+0000 7f2ca8b4b640 1 librados: shutdown 2024-03-25T18:14:14.252 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new a0e072c8-3cd3-49ab-843e-32eef8e51002 3 2024-03-25T18:14:14.298 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.298+0000 7ff6ff045040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-25T18:14:14.326 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:14.326+0000 7f3c176da040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:14:14.351 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.351+0000 7f7166cb3640 10 monclient: get_monmap_and_config 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.351+0000 7f7166cb3640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.351+0000 7f7166cb3640 10 monclient: monmap: 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:14.352177+0000 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:14.352177+0000 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:14.352 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.353 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.351+0000 7f7166cb3640 1 Processor -- start 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.351+0000 7f7166cb3640 1 -- start start 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.351+0000 7f7166cb3640 10 monclient: init 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f71600b4b50 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f716013ff20 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f7160142dd0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600ab440 con 0x7f71600b4b50 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600ab6b0 con 0x7f716013ff20 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71600a9f10 con 0x7f7160142dd0 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.352+0000 7f7166cb3640 10 monclient(hunting): authenticate will time out at 7298.626338s 2024-03-25T18:14:14.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.353+0000 7f715ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f716013ff20 0x7f71601426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38620/0 (socket says 172.21.15.183:38620) 2024-03-25T18:14:14.355 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.353+0000 7f715ffff640 1 -- 172.21.15.183:0/644636542 learned_addr learned my addr 172.21.15.183:0/644636542 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.355+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1899587314 0 0) 0x7f71600ab440 con 0x7f71600b4b50 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.355+0000 7f715f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.355+0000 7f715f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.355+0000 7f715f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.355+0000 7f715f7fe640 10 monclient(hunting): my global_id is 4128 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.355+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f714c003520 con 0x7f71600b4b50 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4102239894 0 0) 0x7f71600ab6b0 con 0x7f716013ff20 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.357 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): my global_id is 4126 2024-03-25T18:14:14.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71600ab440 con 0x7f716013ff20 2024-03-25T18:14:14.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3995430232 0 0) 0x7f71600a9f10 con 0x7f7160142dd0 2024-03-25T18:14:14.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 10 monclient(hunting): my global_id is 4142 2024-03-25T18:14:14.358 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.356+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71600ab6b0 con 0x7f7160142dd0 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.358+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2202627049 0 0) 0x7f714c003520 con 0x7f71600b4b50 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.358+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 >> v1:172.21.15.7:6790/0 conn(0x7f7160142dd0 legacy=0x7f716014df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 >> v1:172.21.15.183:6789/0 conn(0x7f716013ff20 legacy=0x7f71601426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71600aa0a0 con 0x7f71600b4b50 2024-03-25T18:14:14.360 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f7166cb3640 5 monclient: authenticate success, global_id 4128 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f7166cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7158001650 con 0x7f71600b4b50 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f715f7fe640 10 monclient: dump: 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.361 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.362 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.362 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.359+0000 7f7166cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:14.363 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7158001830 con 0x7f71600b4b50 2024-03-25T18:14:14.363 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f715f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:14.363 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f715f7fe640 1 -- 172.21.15.183:0/644636542 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7158001bc0 con 0x7f71600b4b50 2024-03-25T18:14:14.363 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f715f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f715f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f715f7fe640 10 monclient: dump: 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f7166cb3640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.360+0000 7f7166cb3640 4 set_mon_vals no callback set 2024-03-25T18:14:14.364 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.361+0000 7f7166cb3640 10 monclient: shutdown 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.361+0000 7f7166cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.361+0000 7f7166cb3640 1 -- 172.21.15.183:0/644636542 >> v1:172.21.15.7:6789/0 conn(0x7f71600b4b50 legacy=0x7f716013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.361+0000 7f7166cb3640 1 -- 172.21.15.183:0/644636542 shutdown_connections 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.361+0000 7f7166cb3640 1 -- 172.21.15.183:0/644636542 >> 172.21.15.183:0/644636542 conn(0x7f7160066920 msgr2=0x7f7160066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.361+0000 7f7166cb3640 1 -- 172.21.15.183:0/644636542 shutdown_connections 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.361+0000 7f7166cb3640 1 -- 172.21.15.183:0/644636542 wait complete. 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 10 monclient: monmap: 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:14.365 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 1 librados: starting msgr at 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 1 librados: starting objecter 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 1 Processor -- start 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 1 -- start start 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 1 librados: setting wanted keys 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 1 librados: calling monclient init 2024-03-25T18:14:14.366 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.362+0000 7f7166cb3640 10 monclient: init 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f71600b4b50 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f716013ff20 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f7160142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7160145130 con 0x7f7160142dd0 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71601dbd40 con 0x7f71600b4b50 2024-03-25T18:14:14.368 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71601dba80 con 0x7f716013ff20 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f7166cb3640 10 monclient(hunting): authenticate will time out at 7298.637300s 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f715ffff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f716013ff20 0x7f71600b1120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.183:37670/0 (socket says 172.21.15.183:37670) 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.363+0000 7f715ffff640 1 -- 172.21.15.183:0/338298408 learned_addr learned my addr 172.21.15.183:0/338298408 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.364+0000 7f715d7fa640 10 client.?.objecter ms_handle_connect 0x7f71600b4b50 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.364+0000 7f715d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.364+0000 7f715d7fa640 10 client.?.objecter ms_handle_connect 0x7f716013ff20 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.364+0000 7f715d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.364+0000 7f715d7fa640 10 client.?.objecter ms_handle_connect 0x7f7160142dd0 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.364+0000 7f715d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3523600946 0 0) 0x7f71601dbd40 con 0x7f71600b4b50 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.369 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): my global_id is 4129 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7140004eb0 con 0x7f71600b4b50 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780938198 0 0) 0x7f7160145130 con 0x7f7160142dd0 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 10 monclient(hunting): my global_id is 4131 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.366+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71601dbd40 con 0x7f7160142dd0 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.367+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 254897416 0 0) 0x7f71601dba80 con 0x7f716013ff20 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.367+0000 7f715d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.367+0000 7f715d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.367+0000 7f715d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.367+0000 7f715d7fa640 10 monclient(hunting): my global_id is 4145 2024-03-25T18:14:14.370 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.367+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7160145130 con 0x7f716013ff20 2024-03-25T18:14:14.371 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118769922 0 0) 0x7f71601dbd40 con 0x7f7160142dd0 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71601dba80 con 0x7f7160142dd0 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f71480017d0 con 0x7f7160142dd0 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 10 monclient(hunting): dump: 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.372 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1477232216 0 0) 0x7f7140004eb0 con 0x7f71600b4b50 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f71601dbd40 con 0x7f71600b4b50 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7158001b40 con 0x7f71600b4b50 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.370+0000 7f715d7fa640 10 monclient(hunting): dump: 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.373 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 10199569 0 0) 0x7f71601dbd40 con 0x7f71600b4b50 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 >> v1:172.21.15.7:6790/0 conn(0x7f716013ff20 legacy=0x7f71600b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 >> v1:172.21.15.7:6789/0 conn(0x7f7160142dd0 legacy=0x7f71601d7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 1 monclient: found mon.b 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:14.375 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71601d5450 con 0x7f71600b4b50 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f715d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 5 monclient: authenticate success, global_id 4129 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 10 monclient: _renew_subs 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71600b1830 con 0x7f71600b4b50 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 10 client.4129.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 10 monclient: _renew_subs 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f716013a290 con 0x7f71600b4b50 2024-03-25T18:14:14.376 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.374+0000 7f7166cb3640 1 librados: init done 2024-03-25T18:14:14.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.375+0000 7f7166cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.375+0000 7f7166cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.375+0000 7f7166cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:14.377 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.375+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716004f730 con 0x7f71600b4b50 2024-03-25T18:14:14.422 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.422+0000 7ff6ff045040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-25T18:14:14.469 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.468+0000 7ff6ff045040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-25T18:14:14.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f71580038a0 con 0x7f71600b4b50 2024-03-25T18:14:14.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:14.473 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7158003bb0 con 0x7f71600b4b50 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 10 monclient: dump: 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f71580040a0 con 0x7f71600b4b50 2024-03-25T18:14:14.474 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715effd640 4 set_mon_vals no callback set 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 2171291838 0 0) 0x7f71580056b0 con 0x7f71600b4b50 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 10 client.4129.objecter ms_dispatch 0x7f7160000b90 osd_map(5..5 src has 1..5) v4 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 3 client.4129.objecter handle_osd_map got epochs [5,5] > 0 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 3 client.4129.objecter handle_osd_map decoding full epoch 5 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 20 client.4129.objecter dump_active .. 0 homeless 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f7158005ab0 con 0x7f71600b4b50 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.475 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.472+0000 7f715d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:14.495 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:14.492+0000 7f3c176da040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:14.537 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:14.536+0000 7f3c176da040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:14.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.546+0000 7f7166cb3640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a0e072c8-3cd3-49ab-843e-32eef8e51002", "id": 3}] 2024-03-25T18:14:14.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.546+0000 7f7166cb3640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a0e072c8-3cd3-49ab-843e-32eef8e51002", "id": 3}] 2024-03-25T18:14:14.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.546+0000 7f7166cb3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:14.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.546+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a0e072c8-3cd3-49ab-843e-32eef8e51002", "id": 3} v 0) v1 -- 0x7f71600b5ef0 con 0x7f71600b4b50 2024-03-25T18:14:14.586 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.585+0000 7ff6ff045040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:14.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.641+0000 7f715d7fa640 1 -- 172.21.15.183:0/338298408 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a0e072c8-3cd3-49ab-843e-32eef8e51002", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 3485417542 0 525046306) 0x7f7158005230 con 0x7f71600b4b50 2024-03-25T18:14:14.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.641+0000 7f715d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a0e072c8-3cd3-49ab-843e-32eef8e51002", "id": 3}] 2024-03-25T18:14:14.642 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.641+0000 7f715d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:14.642 INFO:teuthology.orchestra.run.smithi183.stdout:3 2024-03-25T18:14:14.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.641+0000 7f7166cb3640 10 librados: watch_flush enter 2024-03-25T18:14:14.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.641+0000 7f7166cb3640 10 librados: watch_flush exit 2024-03-25T18:14:14.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.641+0000 7f7166cb3640 20 client.4129.objecter shutdown clearing up homeless session... 2024-03-25T18:14:14.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.641+0000 7f7166cb3640 10 client.4129.objecter successfully canceled tick 2024-03-25T18:14:14.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 10 monclient: shutdown 2024-03-25T18:14:14.643 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:14.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 >> v1:172.21.15.183:6789/0 conn(0x7f71600b4b50 legacy=0x7f71601db370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 shutdown_connections 2024-03-25T18:14:14.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 >> 172.21.15.183:0/338298408 conn(0x7f7160066920 msgr2=0x7f71600a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:14.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 shutdown_connections 2024-03-25T18:14:14.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 1 -- 172.21.15.183:0/338298408 wait complete. 2024-03-25T18:14:14.644 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.642+0000 7f7166cb3640 1 librados: shutdown 2024-03-25T18:14:14.653 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new d580d02b-51de-41f1-90c1-60cfcb18553e 4 2024-03-25T18:14:14.689 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.689+0000 7ff6ff045040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-25T18:14:14.712 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:14.711+0000 7f3c176da040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:14:14.752 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.751+0000 7ff48c282640 10 monclient: get_monmap_and_config 2024-03-25T18:14:14.752 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.751+0000 7ff48c282640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.752 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.751+0000 7ff48c282640 10 monclient: monmap: 2024-03-25T18:14:14.752 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:14.752 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.752 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:14.752710+0000 2024-03-25T18:14:14.753 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:14.752710+0000 2024-03-25T18:14:14.753 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.753 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.753 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:14.753 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:14.753 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:14.753 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.754 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.752+0000 7ff48c282640 1 Processor -- start 2024-03-25T18:14:14.754 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.752+0000 7ff48c282640 1 -- start start 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.752+0000 7ff48c282640 10 monclient: init 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 10 monclient(hunting): picked mon.noname-c con 0x7ff4840b4b50 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 10 monclient(hunting): picked mon.noname-a con 0x7ff48413ff20 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 10 monclient(hunting): picked mon.noname-b con 0x7ff484142dd0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840ab440 con 0x7ff48413ff20 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840ab6b0 con 0x7ff4840b4b50 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4840a9f10 con 0x7ff484142dd0 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48c282640 10 monclient(hunting): authenticate will time out at 7299.026934s 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48a01e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff4840b4b50 0x7ff48413f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38640/0 (socket says 172.21.15.183:38640) 2024-03-25T18:14:14.755 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.753+0000 7ff48a01e640 1 -- 172.21.15.183:0/778133965 learned_addr learned my addr 172.21.15.183:0/778133965 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:14.757 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 858254935 0 0) 0x7ff4840ab6b0 con 0x7ff4840b4b50 2024-03-25T18:14:14.757 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.757 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.757 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.757 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): my global_id is 4132 2024-03-25T18:14:14.758 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff470003520 con 0x7ff4840b4b50 2024-03-25T18:14:14.758 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242837320 0 0) 0x7ff4840a9f10 con 0x7ff484142dd0 2024-03-25T18:14:14.758 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.758 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.758 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.758 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 10 monclient(hunting): my global_id is 4148 2024-03-25T18:14:14.758 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.756+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4840ab6b0 con 0x7ff484142dd0 2024-03-25T18:14:14.759 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.757+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2713007460 0 0) 0x7ff4840ab440 con 0x7ff48413ff20 2024-03-25T18:14:14.759 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.757+0000 7ff48901c640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.759 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.757+0000 7ff48901c640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.759 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.757+0000 7ff48901c640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.759 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.757+0000 7ff48901c640 10 monclient(hunting): my global_id is 4134 2024-03-25T18:14:14.759 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.757+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4840a9f10 con 0x7ff48413ff20 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4209142913 0 0) 0x7ff470003520 con 0x7ff4840b4b50 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 >> v1:172.21.15.7:6790/0 conn(0x7ff484142dd0 legacy=0x7ff48414df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 >> v1:172.21.15.7:6789/0 conn(0x7ff48413ff20 legacy=0x7ff4841426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 1 monclient: found mon.noname-c 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4840aa0a0 con 0x7ff4840b4b50 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff480001690 con 0x7ff4840b4b50 2024-03-25T18:14:14.762 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48c282640 5 monclient: authenticate success, global_id 4132 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48c282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48901c640 10 monclient: dump: 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.763 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.761+0000 7ff48c282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff4800017f0 con 0x7ff4840b4b50 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48901c640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48901c640 1 -- 172.21.15.183:0/778133965 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff480001b80 con 0x7ff4840b4b50 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48901c640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48901c640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48901c640 10 monclient: dump: 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.764 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.765 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.765 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.765 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.765 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.765 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48c282640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:14.765 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48c282640 4 set_mon_vals no callback set 2024-03-25T18:14:14.767 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48c282640 10 monclient: shutdown 2024-03-25T18:14:14.767 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48c282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:14.767 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.763+0000 7ff48c282640 1 -- 172.21.15.183:0/778133965 >> v1:172.21.15.183:6789/0 conn(0x7ff4840b4b50 legacy=0x7ff48413f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.767 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.764+0000 7ff48c282640 1 -- 172.21.15.183:0/778133965 shutdown_connections 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.764+0000 7ff48c282640 1 -- 172.21.15.183:0/778133965 >> 172.21.15.183:0/778133965 conn(0x7ff484066920 msgr2=0x7ff484066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.764+0000 7ff48c282640 1 -- 172.21.15.183:0/778133965 shutdown_connections 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.764+0000 7ff48c282640 1 -- 172.21.15.183:0/778133965 wait complete. 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.764+0000 7ff48c282640 10 monclient: build_initial_monmap 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.764+0000 7ff48c282640 10 monclient: monmap: 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.768 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 librados: starting msgr at 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 librados: starting objecter 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 Processor -- start 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 -- start start 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 librados: setting wanted keys 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 librados: calling monclient init 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient: init 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient(hunting): picked mon.noname-c con 0x7ff4840b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient(hunting): picked mon.noname-b con 0x7ff48413ff20 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient(hunting): picked mon.noname-a con 0x7ff484142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4841447f0 con 0x7ff484142dd0 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff484144ad0 con 0x7ff48413ff20 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff484140430 con 0x7ff4840b4b50 2024-03-25T18:14:14.769 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.765+0000 7ff48c282640 10 monclient(hunting): authenticate will time out at 7299.039693s 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff48981d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff48413ff20 0x7ff484144ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38644/0 (socket says 172.21.15.183:38644) 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff48981d640 1 -- 172.21.15.183:0/3771516447 learned_addr learned my addr 172.21.15.183:0/3771516447 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff47affd640 10 client.?.objecter ms_handle_connect 0x7ff48413ff20 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff47affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff47affd640 10 client.?.objecter ms_handle_connect 0x7ff484142dd0 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff47affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff47affd640 10 client.?.objecter ms_handle_connect 0x7ff4840b4b50 2024-03-25T18:14:14.770 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.766+0000 7ff47affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318130484 0 0) 0x7ff484144ad0 con 0x7ff48413ff20 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): my global_id is 4135 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff468004eb0 con 0x7ff48413ff20 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929827543 0 0) 0x7ff4841447f0 con 0x7ff484142dd0 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.771 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): my global_id is 4137 2024-03-25T18:14:14.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff484144ad0 con 0x7ff484142dd0 2024-03-25T18:14:14.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2809501462 0 0) 0x7ff484140430 con 0x7ff4840b4b50 2024-03-25T18:14:14.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:14.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:14.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:14.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 10 monclient(hunting): my global_id is 4151 2024-03-25T18:14:14.772 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.769+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff4841447f0 con 0x7ff4840b4b50 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 315637131 0 0) 0x7ff484144ad0 con 0x7ff484142dd0 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff484140430 con 0x7ff484142dd0 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff47c0018f0 con 0x7ff484142dd0 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 10 monclient(hunting): dump: 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.777 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2108175233 0 0) 0x7ff468004eb0 con 0x7ff48413ff20 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff484144ad0 con 0x7ff48413ff20 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff474001a10 con 0x7ff48413ff20 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.775+0000 7ff47affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 10 monclient(hunting): dump: 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.778 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2245195172 0 0) 0x7ff4841447f0 con 0x7ff4840b4b50 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff468004eb0 con 0x7ff4840b4b50 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff480001960 con 0x7ff4840b4b50 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.776+0000 7ff47affd640 10 monclient(hunting): dump: 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.779 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.780 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.780 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.780 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.780 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.780 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2692516120 0 0) 0x7ff484140430 con 0x7ff484142dd0 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 >> v1:172.21.15.7:6790/0 conn(0x7ff4840b4b50 legacy=0x7ff484141a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 >> v1:172.21.15.183:6789/0 conn(0x7ff48413ff20 legacy=0x7ff484144ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 1 monclient: found mon.a 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:14.781 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff484142190 con 0x7ff484142dd0 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 10 monclient: _finish_auth 0 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff47affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 5 monclient: authenticate success, global_id 4137 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 10 monclient: _renew_subs 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff484141450 con 0x7ff484142dd0 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 10 client.4137.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 10 monclient: _renew_subs 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4841457e0 con 0x7ff484142dd0 2024-03-25T18:14:14.782 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.780+0000 7ff48c282640 1 librados: init done 2024-03-25T18:14:14.783 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.781+0000 7ff48c282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.783 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.781+0000 7ff48c282640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.783 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.781+0000 7ff48c282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:14.783 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.781+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff484142430 con 0x7ff484142dd0 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff47c001f60 con 0x7ff484142dd0 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff47c003b10 con 0x7ff484142dd0 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 10 monclient: dump: 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.785 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff48881b640 4 set_mon_vals no callback set 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff47c004000 con 0x7ff484142dd0 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2869618717 0 0) 0x7ff47c0056f0 con 0x7ff484142dd0 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 10 client.4137.objecter ms_dispatch 0x7ff484000b90 osd_map(6..6 src has 1..6) v4 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 3 client.4137.objecter handle_osd_map got epochs [6,6] > 0 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 3 client.4137.objecter handle_osd_map decoding full epoch 6 2024-03-25T18:14:14.786 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.784+0000 7ff47affd640 20 client.4137.objecter dump_active .. 0 homeless 2024-03-25T18:14:14.857 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.856+0000 7ff6ff045040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-25T18:14:14.887 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.887+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff47c0052d0 con 0x7ff484142dd0 2024-03-25T18:14:14.887 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.887+0000 7ff47affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:14.888 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.887+0000 7ff47affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:14.900 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:14.899+0000 7ff6ff045040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-25T18:14:14.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.964+0000 7ff48c282640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d580d02b-51de-41f1-90c1-60cfcb18553e", "id": 4}] 2024-03-25T18:14:14.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.964+0000 7ff48c282640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d580d02b-51de-41f1-90c1-60cfcb18553e", "id": 4}] 2024-03-25T18:14:14.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.964+0000 7ff48c282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:14.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:14.964+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d580d02b-51de-41f1-90c1-60cfcb18553e", "id": 4} v 0) v1 -- 0x7ff48404f730 con 0x7ff484142dd0 2024-03-25T18:14:15.035 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.034+0000 7ff47affd640 1 -- 172.21.15.183:0/3771516447 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d580d02b-51de-41f1-90c1-60cfcb18553e", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 1190194585 0 1697012583) 0x7ff47c005920 con 0x7ff484142dd0 2024-03-25T18:14:15.035 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.034+0000 7ff47affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d580d02b-51de-41f1-90c1-60cfcb18553e", "id": 4}] 2024-03-25T18:14:15.035 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.034+0000 7ff47affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:15.035 INFO:teuthology.orchestra.run.smithi183.stdout:4 2024-03-25T18:14:15.036 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 10 librados: watch_flush enter 2024-03-25T18:14:15.036 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 10 librados: watch_flush exit 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 20 client.4137.objecter shutdown clearing up homeless session... 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 10 client.4137.objecter successfully canceled tick 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 10 monclient: shutdown 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 >> v1:172.21.15.7:6789/0 conn(0x7ff484142dd0 legacy=0x7ff4841e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 shutdown_connections 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 >> 172.21.15.183:0/3771516447 conn(0x7ff484066920 msgr2=0x7ff48413f5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 shutdown_connections 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 1 -- 172.21.15.183:0/3771516447 wait complete. 2024-03-25T18:14:15.037 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.035+0000 7ff48c282640 1 librados: shutdown 2024-03-25T18:14:15.046 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new ee928186-6517-45e3-8997-8569eca4afbe 5 2024-03-25T18:14:15.060 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:15.059+0000 7ff6ff045040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.158+0000 7ff64557c640 10 monclient: get_monmap_and_config 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.158+0000 7ff64557c640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.158+0000 7ff64557c640 10 monclient: monmap: 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:15.159571+0000 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:15.159571+0000 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.159 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:15.160 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:15.160 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:15.160 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 1 Processor -- start 2024-03-25T18:14:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 1 -- start start 2024-03-25T18:14:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient: init 2024-03-25T18:14:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient(hunting): picked mon.noname-c con 0x7ff6400b4f00 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:15.161 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient(hunting): picked mon.noname-a con 0x7ff6400b4510 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient(hunting): picked mon.noname-b con 0x7ff640142ad0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400a9d90 con 0x7ff6400b4510 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6400aa2b0 con 0x7ff6400b4f00 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff64013fc60 con 0x7ff640142ad0 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.159+0000 7ff64557c640 10 monclient(hunting): authenticate will time out at 7299.433642s 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.160+0000 7ff63effd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff6400b4f00 0x7ff6401423c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38658/0 (socket says 172.21.15.183:38658) 2024-03-25T18:14:15.162 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.160+0000 7ff63effd640 1 -- 172.21.15.183:0/506441216 learned_addr learned my addr 172.21.15.183:0/506441216 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.162+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 635940880 0 0) 0x7ff6400aa2b0 con 0x7ff6400b4f00 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.162+0000 7ff63dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.162+0000 7ff63dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.162+0000 7ff63dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.162+0000 7ff63dffb640 10 monclient(hunting): my global_id is 4138 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.162+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff628003520 con 0x7ff6400b4f00 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 964841237 0 0) 0x7ff6400a9d90 con 0x7ff6400b4510 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): my global_id is 4140 2024-03-25T18:14:15.164 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6400aa2b0 con 0x7ff6400b4510 2024-03-25T18:14:15.165 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 789737873 0 0) 0x7ff64013fc60 con 0x7ff640142ad0 2024-03-25T18:14:15.165 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.165 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.165 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.165 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 10 monclient(hunting): my global_id is 4154 2024-03-25T18:14:15.165 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.163+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6400a9d90 con 0x7ff640142ad0 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.165+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3052734658 0 0) 0x7ff628003520 con 0x7ff6400b4f00 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.165+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 >> v1:172.21.15.7:6790/0 conn(0x7ff640142ad0 legacy=0x7ff64014dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.165+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 >> v1:172.21.15.7:6789/0 conn(0x7ff6400b4510 legacy=0x7ff64013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.165+0000 7ff63dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.165+0000 7ff63dffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.165+0000 7ff63dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff64013fdf0 con 0x7ff6400b4f00 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff630001690 con 0x7ff6400b4f00 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.167 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff63dffb640 10 monclient: dump: 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff64557c640 5 monclient: authenticate success, global_id 4138 2024-03-25T18:14:15.168 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.166+0000 7ff64557c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:15.171 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6300017f0 con 0x7ff6400b4f00 2024-03-25T18:14:15.171 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff63dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.171 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff63dffb640 1 -- 172.21.15.183:0/506441216 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff630001c00 con 0x7ff6400b4f00 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff64557c640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff64557c640 4 set_mon_vals no callback set 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff63dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff63dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.167+0000 7ff63dffb640 10 monclient: dump: 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.172 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.168+0000 7ff64557c640 10 monclient: shutdown 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.168+0000 7ff64557c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.168+0000 7ff64557c640 1 -- 172.21.15.183:0/506441216 >> v1:172.21.15.183:6789/0 conn(0x7ff6400b4f00 legacy=0x7ff6401423c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.168+0000 7ff64557c640 1 -- 172.21.15.183:0/506441216 shutdown_connections 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.168+0000 7ff64557c640 1 -- 172.21.15.183:0/506441216 >> 172.21.15.183:0/506441216 conn(0x7ff640066b00 msgr2=0x7ff640066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.168+0000 7ff64557c640 1 -- 172.21.15.183:0/506441216 shutdown_connections 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.168+0000 7ff64557c640 1 -- 172.21.15.183:0/506441216 wait complete. 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 10 monclient: monmap: 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:15.173 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 1 librados: starting msgr at 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 1 librados: starting objecter 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 1 Processor -- start 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 1 -- start start 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 1 librados: setting wanted keys 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 1 librados: calling monclient init 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.169+0000 7ff64557c640 10 monclient: init 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 10 monclient(hunting): picked mon.noname-a con 0x7ff6400b4510 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 10 monclient(hunting): picked mon.noname-b con 0x7ff6400b4f00 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 10 monclient(hunting): picked mon.noname-c con 0x7ff640142ad0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6401cd830 con 0x7ff6400b4510 2024-03-25T18:14:15.174 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6401cc090 con 0x7ff6400b4f00 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6401cc640 con 0x7ff640142ad0 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff64557c640 10 monclient(hunting): authenticate will time out at 7299.444033s 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff63e7fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff6400b4f00 0x7ff640137f90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38672/0 (socket says 172.21.15.183:38672) 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff63e7fc640 1 -- 172.21.15.183:0/1454520257 learned_addr learned my addr 172.21.15.183:0/1454520257 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.170+0000 7ff6277fe640 10 client.?.objecter ms_handle_connect 0x7ff6400b4f00 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.171+0000 7ff6277fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.171+0000 7ff6277fe640 10 client.?.objecter ms_handle_connect 0x7ff6400b4510 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.171+0000 7ff6277fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.171+0000 7ff6277fe640 10 client.?.objecter ms_handle_connect 0x7ff640142ad0 2024-03-25T18:14:15.175 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.171+0000 7ff6277fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.176 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 592056821 0 0) 0x7ff6401cd830 con 0x7ff6400b4510 2024-03-25T18:14:15.176 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.176 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.176 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.176 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): my global_id is 4143 2024-03-25T18:14:15.176 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff62c004eb0 con 0x7ff6400b4510 2024-03-25T18:14:15.177 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2245721748 0 0) 0x7ff6401cc090 con 0x7ff6400b4f00 2024-03-25T18:14:15.177 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 10 monclient(hunting): my global_id is 4141 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.172+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6401cd830 con 0x7ff6400b4f00 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.173+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 465004590 0 0) 0x7ff6401cc640 con 0x7ff640142ad0 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.173+0000 7ff6277fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.173+0000 7ff6277fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.173+0000 7ff6277fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.173+0000 7ff6277fe640 10 monclient(hunting): my global_id is 4157 2024-03-25T18:14:15.178 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.173+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6401cc090 con 0x7ff640142ad0 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.177+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2860360052 0 0) 0x7ff6401cd830 con 0x7ff6400b4f00 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.177+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6401cc640 con 0x7ff6400b4f00 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.177+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff634001a50 con 0x7ff6400b4f00 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.177+0000 7ff6277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.177+0000 7ff6277fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.177+0000 7ff6277fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.180 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.177+0000 7ff6277fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.178+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3505136 0 0) 0x7ff6401cc090 con 0x7ff640142ad0 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.178+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6401cd830 con 0x7ff640142ad0 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.178+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff638001bd0 con 0x7ff640142ad0 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.178+0000 7ff6277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.178+0000 7ff6277fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.178+0000 7ff6277fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.181 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.182 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.182 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.182 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.182 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.182 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.180+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1758808916 0 0) 0x7ff62c004eb0 con 0x7ff6400b4510 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.180+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6401cc090 con 0x7ff6400b4510 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.180+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff630001b00 con 0x7ff6400b4510 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.180+0000 7ff6277fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.180+0000 7ff6277fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.180+0000 7ff6277fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.184 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 167937439 0 0) 0x7ff6401cc640 con 0x7ff6400b4f00 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 >> v1:172.21.15.7:6790/0 conn(0x7ff640142ad0 legacy=0x7ff6401d7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 >> v1:172.21.15.7:6789/0 conn(0x7ff6400b4510 legacy=0x7ff6401cb8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 1 monclient: found mon.b 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6400b8960 con 0x7ff6400b4f00 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.185 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff6277fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 5 monclient: authenticate success, global_id 4141 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 10 monclient: _renew_subs 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6400b8c10 con 0x7ff6400b4f00 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 10 client.4141.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 10 monclient: _renew_subs 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff640138880 con 0x7ff6400b4f00 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.181+0000 7ff64557c640 1 librados: init done 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.182+0000 7ff64557c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.182+0000 7ff64557c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.182+0000 7ff64557c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.186 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.182+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6401cc220 con 0x7ff6400b4f00 2024-03-25T18:14:15.281 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff634001bb0 con 0x7ff6400b4f00 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff634003a60 con 0x7ff6400b4f00 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 10 monclient: dump: 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.282 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7ff634004010 con 0x7ff6400b4f00 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff63d7fa640 4 set_mon_vals no callback set 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 2907480073 0 0) 0x7ff634006810 con 0x7ff6400b4f00 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 10 client.4141.objecter ms_dispatch 0x7ff640000b90 osd_map(7..7 src has 1..7) v4 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 3 client.4141.objecter handle_osd_map got epochs [7,7] > 0 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 3 client.4141.objecter handle_osd_map decoding full epoch 7 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 20 client.4141.objecter dump_active .. 0 homeless 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7ff634018e00 con 0x7ff6400b4f00 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.283 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.280+0000 7ff6277fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:15.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.353+0000 7ff64557c640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ee928186-6517-45e3-8997-8569eca4afbe", "id": 5}] 2024-03-25T18:14:15.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.353+0000 7ff64557c640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ee928186-6517-45e3-8997-8569eca4afbe", "id": 5}] 2024-03-25T18:14:15.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.353+0000 7ff64557c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.354 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.353+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ee928186-6517-45e3-8997-8569eca4afbe", "id": 5} v 0) v1 -- 0x7ff6400ab100 con 0x7ff6400b4f00 2024-03-25T18:14:15.427 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.426+0000 7ff6277fe640 1 -- 172.21.15.183:0/1454520257 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ee928186-6517-45e3-8997-8569eca4afbe", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 1615202914 0 1988412176) 0x7ff6340042c0 con 0x7ff6400b4f00 2024-03-25T18:14:15.427 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.426+0000 7ff6277fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ee928186-6517-45e3-8997-8569eca4afbe", "id": 5}] 2024-03-25T18:14:15.427 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.426+0000 7ff6277fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:15.427 INFO:teuthology.orchestra.run.smithi183.stdout:5 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 10 librados: watch_flush enter 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 10 librados: watch_flush exit 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 20 client.4141.objecter shutdown clearing up homeless session... 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 10 client.4141.objecter successfully canceled tick 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 10 monclient: shutdown 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 >> v1:172.21.15.183:6789/0 conn(0x7ff6400b4f00 legacy=0x7ff640137f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 shutdown_connections 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 >> 172.21.15.183:0/1454520257 conn(0x7ff640066b00 msgr2=0x7ff64013b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 shutdown_connections 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 1 -- 172.21.15.183:0/1454520257 wait complete. 2024-03-25T18:14:15.429 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.427+0000 7ff64557c640 1 librados: shutdown 2024-03-25T18:14:15.439 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new d514f04c-85ce-455d-8b9c-101dd208eb60 6 2024-03-25T18:14:15.539 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.538+0000 7f63dd332640 10 monclient: get_monmap_and_config 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.538+0000 7f63dd332640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.538+0000 7f63dd332640 10 monclient: monmap: 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:15.539966+0000 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:15.539966+0000 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:15.540 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.539+0000 7f63dd332640 1 Processor -- start 2024-03-25T18:14:15.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.539+0000 7f63dd332640 1 -- start start 2024-03-25T18:14:15.541 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.539+0000 7f63dd332640 10 monclient: init 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 10 monclient(hunting): picked mon.noname-c con 0x7f63d80b4b50 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 10 monclient(hunting): picked mon.noname-b con 0x7f63d813ff20 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 10 monclient(hunting): picked mon.noname-a con 0x7f63d8142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80ab440 con 0x7f63d8142dd0 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80ab6b0 con 0x7f63d80b4b50 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d80a9f10 con 0x7f63d813ff20 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63dd332640 10 monclient(hunting): authenticate will time out at 7299.814011s 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63d6ffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f63d80b4b50 0x7f63d813f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38688/0 (socket says 172.21.15.183:38688) 2024-03-25T18:14:15.542 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.540+0000 7f63d6ffd640 1 -- 172.21.15.183:0/2190091129 learned_addr learned my addr 172.21.15.183:0/2190091129 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.542+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2829747619 0 0) 0x7f63d80ab6b0 con 0x7f63d80b4b50 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.542+0000 7f63d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.542+0000 7f63d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.542+0000 7f63d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.542+0000 7f63d5ffb640 10 monclient(hunting): my global_id is 4144 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.542+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63c4003520 con 0x7f63d80b4b50 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2814034850 0 0) 0x7f63d80ab440 con 0x7f63d8142dd0 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): my global_id is 4146 2024-03-25T18:14:15.544 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d80ab6b0 con 0x7f63d8142dd0 2024-03-25T18:14:15.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2804272642 0 0) 0x7f63d80a9f10 con 0x7f63d813ff20 2024-03-25T18:14:15.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 10 monclient(hunting): my global_id is 4160 2024-03-25T18:14:15.545 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.543+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d80ab440 con 0x7f63d813ff20 2024-03-25T18:14:15.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3591448122 0 0) 0x7f63c4003520 con 0x7f63d80b4b50 2024-03-25T18:14:15.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 >> v1:172.21.15.7:6790/0 conn(0x7f63d813ff20 legacy=0x7f63d81426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 >> v1:172.21.15.7:6789/0 conn(0x7f63d8142dd0 legacy=0x7f63d814df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 1 monclient: found mon.noname-c 2024-03-25T18:14:15.547 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d80aa0a0 con 0x7f63d80b4b50 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63dd332640 5 monclient: authenticate success, global_id 4144 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f63c0001690 con 0x7f63d80b4b50 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63dd332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63d5ffb640 10 monclient: dump: 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.548 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.549 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.549 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.549 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.549 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.549 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.549 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.546+0000 7f63dd332640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63c00017f0 con 0x7f63d80b4b50 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63d5ffb640 1 -- 172.21.15.183:0/2190091129 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f63c0001b80 con 0x7f63d80b4b50 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63dd332640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63dd332640 4 set_mon_vals no callback set 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63d5ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.547+0000 7f63d5ffb640 10 monclient: dump: 2024-03-25T18:14:15.552 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 10 monclient: shutdown 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 1 -- 172.21.15.183:0/2190091129 >> v1:172.21.15.183:6789/0 conn(0x7f63d80b4b50 legacy=0x7f63d813f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 1 -- 172.21.15.183:0/2190091129 shutdown_connections 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 1 -- 172.21.15.183:0/2190091129 >> 172.21.15.183:0/2190091129 conn(0x7f63d8066920 msgr2=0x7f63d8066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.553 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 1 -- 172.21.15.183:0/2190091129 shutdown_connections 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 1 -- 172.21.15.183:0/2190091129 wait complete. 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.548+0000 7f63dd332640 10 monclient: monmap: 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.554 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.549+0000 7f63dd332640 1 librados: starting msgr at 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.549+0000 7f63dd332640 1 librados: starting objecter 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.549+0000 7f63dd332640 1 Processor -- start 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.549+0000 7f63dd332640 1 -- start start 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.549+0000 7f63dd332640 1 librados: setting wanted keys 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.549+0000 7f63dd332640 1 librados: calling monclient init 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.549+0000 7f63dd332640 10 monclient: init 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 10 monclient(hunting): picked mon.noname-c con 0x7f63d80b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 10 monclient(hunting): picked mon.noname-b con 0x7f63d813ff20 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 10 monclient(hunting): picked mon.noname-a con 0x7f63d8142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8144660 con 0x7f63d8142dd0 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d8144940 con 0x7f63d813ff20 2024-03-25T18:14:15.555 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63d813ac90 con 0x7f63d80b4b50 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63dd332640 10 monclient(hunting): authenticate will time out at 7299.824032s 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63d67fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f63d813ff20 0x7f63d813a580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38704/0 (socket says 172.21.15.183:38704) 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63d67fc640 1 -- 172.21.15.183:0/2752269392 learned_addr learned my addr 172.21.15.183:0/2752269392 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63cf7fe640 10 client.?.objecter ms_handle_connect 0x7f63d813ff20 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.550+0000 7f63cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.551+0000 7f63cf7fe640 10 client.?.objecter ms_handle_connect 0x7f63d8142dd0 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.551+0000 7f63cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.551+0000 7f63cf7fe640 10 client.?.objecter ms_handle_connect 0x7f63d80b4b50 2024-03-25T18:14:15.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.551+0000 7f63cf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.552+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336261893 0 0) 0x7f63d8144940 con 0x7f63d813ff20 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.552+0000 7f63cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.552+0000 7f63cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.552+0000 7f63cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.552+0000 7f63cf7fe640 10 monclient(hunting): my global_id is 4147 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.552+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63b8004eb0 con 0x7f63d813ff20 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1226251927 0 0) 0x7f63d8144660 con 0x7f63d8142dd0 2024-03-25T18:14:15.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): my global_id is 4149 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d8144940 con 0x7f63d8142dd0 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2262856627 0 0) 0x7f63d813ac90 con 0x7f63d80b4b50 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 10 monclient(hunting): my global_id is 4163 2024-03-25T18:14:15.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.553+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63d8144660 con 0x7f63d80b4b50 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3659576168 0 0) 0x7f63b8004eb0 con 0x7f63d813ff20 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63d813ac90 con 0x7f63d813ff20 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f63c8001a10 con 0x7f63d813ff20 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.560 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.557+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2932862866 0 0) 0x7f63d8144940 con 0x7f63d8142dd0 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.558+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63b8004eb0 con 0x7f63d8142dd0 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.558+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f63d0001b50 con 0x7f63d8142dd0 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.558+0000 7f63cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.558+0000 7f63cf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.558+0000 7f63cf7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.561 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.559+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3008613669 0 0) 0x7f63d8144660 con 0x7f63d80b4b50 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.559+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f63d8144940 con 0x7f63d80b4b50 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.559+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f63c0001960 con 0x7f63d80b4b50 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.559+0000 7f63cf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.559+0000 7f63cf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.559+0000 7f63cf7fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.562 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.563 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.563 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.563 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.563 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3517347971 0 0) 0x7f63d813ac90 con 0x7f63d813ff20 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 >> v1:172.21.15.7:6790/0 conn(0x7f63d80b4b50 legacy=0x7f63d8141a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 >> v1:172.21.15.7:6789/0 conn(0x7f63d8142dd0 legacy=0x7f63d81e22f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 1 monclient: found mon.b 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.564 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d81dadb0 con 0x7f63d813ff20 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63cf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 5 monclient: authenticate success, global_id 4147 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 10 monclient: _renew_subs 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d81db000 con 0x7f63d813ff20 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 10 client.4147.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 10 monclient: _renew_subs 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d8139d10 con 0x7f63d813ff20 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.560+0000 7f63dd332640 1 librados: init done 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.561+0000 7f63dd332640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.565 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.561+0000 7f63dd332640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.561+0000 7f63dd332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.561+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d813a2a0 con 0x7f63d813ff20 2024-03-25T18:14:15.566 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63c8001c70 con 0x7f63d813ff20 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f63c80038c0 con 0x7f63d813ff20 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 10 monclient: dump: 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.567 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63d57fa640 4 set_mon_vals no callback set 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f63c8003e70 con 0x7f63d813ff20 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 1852061547 0 0) 0x7f63c80066d0 con 0x7f63d813ff20 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 10 client.4147.objecter ms_dispatch 0x7f63d8000b90 osd_map(8..8 src has 1..8) v4 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 3 client.4147.objecter handle_osd_map got epochs [8,8] > 0 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 3 client.4147.objecter handle_osd_map decoding full epoch 8 2024-03-25T18:14:15.568 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.565+0000 7f63cf7fe640 20 client.4147.objecter dump_active .. 0 homeless 2024-03-25T18:14:15.595 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:15.594+0000 7f3c176da040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:14:15.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.658+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f63c80061f0 con 0x7f63d813ff20 2024-03-25T18:14:15.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.658+0000 7f63cf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.659 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.658+0000 7f63cf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:15.731 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.730+0000 7f63dd332640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "d514f04c-85ce-455d-8b9c-101dd208eb60", "id": 6}] 2024-03-25T18:14:15.732 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.730+0000 7f63dd332640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "d514f04c-85ce-455d-8b9c-101dd208eb60", "id": 6}] 2024-03-25T18:14:15.732 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.731+0000 7f63dd332640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.732 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.731+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "d514f04c-85ce-455d-8b9c-101dd208eb60", "id": 6} v 0) v1 -- 0x7f63d8142160 con 0x7f63d813ff20 2024-03-25T18:14:15.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.804+0000 7f63cf7fe640 1 -- 172.21.15.183:0/2752269392 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "d514f04c-85ce-455d-8b9c-101dd208eb60", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 2483608788 0 1113811849) 0x7f63c8006430 con 0x7f63d813ff20 2024-03-25T18:14:15.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.804+0000 7f63cf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "d514f04c-85ce-455d-8b9c-101dd208eb60", "id": 6}] 2024-03-25T18:14:15.805 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.804+0000 7f63cf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:15.805 INFO:teuthology.orchestra.run.smithi183.stdout:6 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 10 librados: watch_flush enter 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 10 librados: watch_flush exit 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 20 client.4147.objecter shutdown clearing up homeless session... 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 10 client.4147.objecter successfully canceled tick 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 10 monclient: shutdown 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 >> v1:172.21.15.183:6789/0 conn(0x7f63d813ff20 legacy=0x7f63d813a580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 shutdown_connections 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 >> 172.21.15.183:0/2752269392 conn(0x7f63d8066920 msgr2=0x7f63d80b3480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 shutdown_connections 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 1 -- 172.21.15.183:0/2752269392 wait complete. 2024-03-25T18:14:15.807 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.805+0000 7f63dd332640 1 librados: shutdown 2024-03-25T18:14:15.817 DEBUG:teuthology.orchestra.run.smithi183:> sudo ceph --cluster ceph osd new 543897a0-3263-4a41-ba5a-8c3dc035283a 7 2024-03-25T18:14:15.886 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:15.885+0000 7ff6ff045040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-25T18:14:15.894 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:15.893+0000 7f3c176da040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:14:15.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.914+0000 7f8c01489640 10 monclient: get_monmap_and_config 2024-03-25T18:14:15.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.914+0000 7f8c01489640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.915 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.914+0000 7f8c01489640 10 monclient: monmap: 2024-03-25T18:14:15.915 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:15.915 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.915 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:14:15.915824+0000 2024-03-25T18:14:15.916 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:14:15.915824+0000 2024-03-25T18:14:15.916 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.916 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.916 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:15.916 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:15.916 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:15.916 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.917 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.915+0000 7f8c01489640 1 Processor -- start 2024-03-25T18:14:15.917 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.915+0000 7f8c01489640 1 -- start start 2024-03-25T18:14:15.917 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.915+0000 7f8c01489640 10 monclient: init 2024-03-25T18:14:15.917 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 10 monclient(hunting): picked mon.noname-b con 0x7f8bfc0b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 10 monclient(hunting): picked mon.noname-c con 0x7f8bfc13ff20 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 10 monclient(hunting): picked mon.noname-a con 0x7f8bfc142dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0ab440 con 0x7f8bfc142dd0 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0ab6b0 con 0x7f8bfc13ff20 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc0a9f10 con 0x7f8bfc0b4b50 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8c01489640 10 monclient(hunting): authenticate will time out at 7300.190007s 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8bf3fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f8bfc13ff20 0x7f8bfc1426c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38716/0 (socket says 172.21.15.183:38716) 2024-03-25T18:14:15.918 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.916+0000 7f8bf3fff640 1 -- 172.21.15.183:0/3853953990 learned_addr learned my addr 172.21.15.183:0/3853953990 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:15.950 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.950+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237284493 0 0) 0x7f8bfc0ab6b0 con 0x7f8bfc13ff20 2024-03-25T18:14:15.950 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.950+0000 7f8bfa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.950 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.950+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.951 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.950+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.951 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.950+0000 7f8bfa7fc640 10 monclient(hunting): my global_id is 4150 2024-03-25T18:14:15.951 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.950+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8be8003520 con 0x7f8bfc13ff20 2024-03-25T18:14:15.951 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:15.950+0000 7f3c176da040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:14:15.953 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.952+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1223592675 0 0) 0x7f8bfc0a9f10 con 0x7f8bfc0b4b50 2024-03-25T18:14:15.953 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.952+0000 7f8bfa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.953 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.952+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.954 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.952+0000 7f8bfa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.954 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.952+0000 7f8bfa7fc640 10 monclient(hunting): my global_id is 4166 2024-03-25T18:14:15.954 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.952+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc0ab6b0 con 0x7f8bfc0b4b50 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 866680715 0 0) 0x7f8be8003520 con 0x7f8bfc13ff20 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 >> v1:172.21.15.7:6790/0 conn(0x7f8bfc0b4b50 legacy=0x7f8bfc13f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 >> v1:172.21.15.7:6789/0 conn(0x7f8bfc142dd0 legacy=0x7f8bfc14df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.955 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc0aa0a0 con 0x7f8bfc13ff20 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8be4001690 con 0x7f8bfc13ff20 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8bfa7fc640 10 monclient: dump: 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.956 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.957 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8c01489640 5 monclient: authenticate success, global_id 4150 2024-03-25T18:14:15.957 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.953+0000 7f8c01489640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:15.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8be40017f0 con 0x7f8bfc13ff20 2024-03-25T18:14:15.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8bfa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8bfa7fc640 1 -- 172.21.15.183:0/3853953990 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8be4001b80 con 0x7f8bfc13ff20 2024-03-25T18:14:15.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8bfa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8bfa7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.959 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8bfa7fc640 10 monclient: dump: 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8c01489640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8c01489640 4 set_mon_vals no callback set 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8c01489640 10 monclient: shutdown 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8c01489640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8c01489640 1 -- 172.21.15.183:0/3853953990 >> v1:172.21.15.183:6789/0 conn(0x7f8bfc13ff20 legacy=0x7f8bfc1426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8c01489640 1 -- 172.21.15.183:0/3853953990 shutdown_connections 2024-03-25T18:14:15.960 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.955+0000 7f8c01489640 1 -- 172.21.15.183:0/3853953990 >> 172.21.15.183:0/3853953990 conn(0x7f8bfc066920 msgr2=0x7f8bfc066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.956+0000 7f8c01489640 1 -- 172.21.15.183:0/3853953990 shutdown_connections 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.956+0000 7f8c01489640 1 -- 172.21.15.183:0/3853953990 wait complete. 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.956+0000 7f8c01489640 10 monclient: build_initial_monmap 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.956+0000 7f8c01489640 10 monclient: monmap: 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 0 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 0.000000 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:created 0.000000 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 1 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.956+0000 7f8c01489640 1 librados: starting msgr at 2024-03-25T18:14:15.961 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.956+0000 7f8c01489640 1 librados: starting objecter 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 1 Processor -- start 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 1 -- start start 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 1 librados: setting wanted keys 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 1 librados: calling monclient init 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient: init 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient(hunting): picked mon.noname-c con 0x7f8bfc0b4b50 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient(hunting): picked mon.noname-a con 0x7f8bfc13ff20 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient(hunting): picked mon.noname-b con 0x7f8bfc142dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc140850 con 0x7f8bfc13ff20 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc1414a0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bfc1410e0 con 0x7f8bfc0b4b50 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:15.962 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.957+0000 7f8c01489640 10 monclient(hunting): authenticate will time out at 7300.231582s 2024-03-25T18:14:15.963 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.959+0000 7f8bfb7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f8bfc142dd0 0x7f8bfc1dad10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.183:38728/0 (socket says 172.21.15.183:38728) 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.959+0000 7f8bfb7fe640 1 -- 172.21.15.183:0/2129188315 learned_addr learned my addr 172.21.15.183:0/2129188315 (peer_addr_for_me v1:172.21.15.183:0/0) 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.959+0000 7f8bf37fe640 10 client.?.objecter ms_handle_connect 0x7f8bfc142dd0 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.959+0000 7f8bf37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.959+0000 7f8bf37fe640 10 client.?.objecter ms_handle_connect 0x7f8bfc13ff20 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.959+0000 7f8bf37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.961+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3132613887 0 0) 0x7f8bfc1414a0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): my global_id is 4153 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bdc004eb0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 400046635 0 0) 0x7f8bfc140850 con 0x7f8bfc13ff20 2024-03-25T18:14:15.964 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 10 monclient(hunting): my global_id is 4155 2024-03-25T18:14:15.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.962+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc1414a0 con 0x7f8bfc13ff20 2024-03-25T18:14:15.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.963+0000 7f8bf37fe640 10 client.?.objecter ms_handle_connect 0x7f8bfc0b4b50 2024-03-25T18:14:15.965 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.963+0000 7f8bf37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:15.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.965+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211846821 0 0) 0x7f8bfc1410e0 con 0x7f8bfc0b4b50 2024-03-25T18:14:15.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.965+0000 7f8bf37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:14:15.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.965+0000 7f8bf37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:15.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.965+0000 7f8bf37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:15.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.965+0000 7f8bf37fe640 10 monclient(hunting): my global_id is 4169 2024-03-25T18:14:15.966 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.965+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bfc140850 con 0x7f8bfc0b4b50 2024-03-25T18:14:15.967 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.966+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2496432202 0 0) 0x7f8bdc004eb0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.967 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.966+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bfc1410e0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.967 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.966+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8bf4001c30 con 0x7f8bfc142dd0 2024-03-25T18:14:15.967 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.966+0000 7f8bf37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.966+0000 7f8bf37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.966+0000 7f8bf37fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.968 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.966+0000 7f8bf37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.967+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3143448206 0 0) 0x7f8bfc1414a0 con 0x7f8bfc13ff20 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.967+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8bdc004eb0 con 0x7f8bfc13ff20 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.967+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8be4001b00 con 0x7f8bfc13ff20 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.967+0000 7f8bf37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.967+0000 7f8bf37fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.967+0000 7f8bf37fe640 10 monclient(hunting): dump: 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.969 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.970 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2304786622 0 0) 0x7f8bfc1410e0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 >> v1:172.21.15.7:6790/0 conn(0x7f8bfc0b4b50 legacy=0x7f8bfc1344c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 >> v1:172.21.15.7:6789/0 conn(0x7f8bfc13ff20 legacy=0x7f8bfc141b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:15.971 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 1 monclient: found mon.b 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bfc141270 con 0x7f8bfc142dd0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 10 monclient: _finish_auth 0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8bf37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 5 monclient: authenticate success, global_id 4153 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 10 monclient: _renew_subs 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bfc134c30 con 0x7f8bfc142dd0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 10 client.4153.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 10 monclient: _renew_subs 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bfc142400 con 0x7f8bfc142dd0 2024-03-25T18:14:15.972 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.970+0000 7f8c01489640 1 librados: init done 2024-03-25T18:14:15.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.971+0000 7f8c01489640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.971+0000 7f8c01489640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:15.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.971+0000 7f8c01489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:15.973 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.971+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bfc1416e0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.975 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8bf4003990 con 0x7f8bfc142dd0 2024-03-25T18:14:15.975 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:15.975 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8bf4003b80 con 0x7f8bfc142dd0 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 10 monclient: dump: 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:epoch 1 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:election_strategy: 3 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f8bf4004130 con 0x7f8bfc142dd0 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 2193586094 0 0) 0x7f8bf40069c0 con 0x7f8bfc142dd0 2024-03-25T18:14:15.976 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 10 client.4153.objecter ms_dispatch 0x7f8bfc000b90 osd_map(9..9 src has 1..9) v4 2024-03-25T18:14:15.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf9ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:15.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 3 client.4153.objecter handle_osd_map got epochs [9,9] > 0 2024-03-25T18:14:15.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 3 client.4153.objecter handle_osd_map decoding full epoch 9 2024-03-25T18:14:15.977 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:15.974+0000 7f8bf37fe640 20 client.4153.objecter dump_active .. 0 homeless 2024-03-25T18:14:16.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.065+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f8bf4005b40 con 0x7f8bfc142dd0 2024-03-25T18:14:16.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.065+0000 7f8bf37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:16.066 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.065+0000 7f8bf37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:16.073 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:16.072+0000 7f3c176da040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:14:16.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.137+0000 7f8c01489640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "543897a0-3263-4a41-ba5a-8c3dc035283a", "id": 7}] 2024-03-25T18:14:16.137 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.137+0000 7f8c01489640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "543897a0-3263-4a41-ba5a-8c3dc035283a", "id": 7}] 2024-03-25T18:14:16.138 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.137+0000 7f8c01489640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:16.138 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.137+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd new", "uuid": "543897a0-3263-4a41-ba5a-8c3dc035283a", "id": 7} v 0) v1 -- 0x7f8bfc04f730 con 0x7f8bfc142dd0 2024-03-25T18:14:16.173 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:16.172+0000 7ff6ff045040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-25T18:14:16.224 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:16.223+0000 7ff6ff045040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-25T18:14:16.339 INFO:tasks.ceph.mgr.x.smithi183.stderr:2024-03-25T18:14:16.339+0000 7ff6ff045040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-25T18:14:16.367 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.366+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3100637744 0 0) 0x7f8bf40043e0 con 0x7f8bfc142dd0 2024-03-25T18:14:16.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.555+0000 7f8bf37fe640 1 -- 172.21.15.183:0/2129188315 <== mon.1 v1:172.21.15.183:6789/0 11 ==== mon_command_ack([{"prefix": "osd new", "uuid": "543897a0-3263-4a41-ba5a-8c3dc035283a", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 338005945 0 1371665406) 0x7f8bf4005ce0 con 0x7f8bfc142dd0 2024-03-25T18:14:16.556 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.555+0000 7f8bf37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "543897a0-3263-4a41-ba5a-8c3dc035283a", "id": 7}] 2024-03-25T18:14:16.557 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.555+0000 7f8bf37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:16.557 INFO:teuthology.orchestra.run.smithi183.stdout:7 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 10 librados: watch_flush enter 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 10 librados: watch_flush exit 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 20 client.4153.objecter shutdown clearing up homeless session... 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 10 client.4153.objecter successfully canceled tick 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 10 monclient: shutdown 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:16.558 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 >> v1:172.21.15.183:6789/0 conn(0x7f8bfc142dd0 legacy=0x7f8bfc1dad10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 shutdown_connections 2024-03-25T18:14:16.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 >> 172.21.15.183:0/2129188315 conn(0x7f8bfc066920 msgr2=0x7f8bfc0a9660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:16.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 shutdown_connections 2024-03-25T18:14:16.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.556+0000 7f8c01489640 1 -- 172.21.15.183:0/2129188315 wait complete. 2024-03-25T18:14:16.559 INFO:teuthology.orchestra.run.smithi183.stderr:2024-03-25T18:14:16.557+0000 7f8c01489640 1 librados: shutdown 2024-03-25T18:14:16.568 DEBUG:tasks.ceph_manager:running osd.0 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.0.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.568 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-25T18:14:16.569 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0 2024-03-25T18:14:16.571 INFO:tasks.ceph.osd.0:Started 2024-03-25T18:14:16.571 DEBUG:tasks.ceph_manager:running osd.1 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.1.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.571 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-25T18:14:16.571 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1 2024-03-25T18:14:16.573 INFO:tasks.ceph.osd.1:Started 2024-03-25T18:14:16.573 DEBUG:tasks.ceph_manager:running osd.2 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.2.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.573 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-25T18:14:16.573 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2 2024-03-25T18:14:16.575 INFO:tasks.ceph.osd.2:Started 2024-03-25T18:14:16.575 DEBUG:tasks.ceph_manager:running osd.3 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.3.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.575 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-25T18:14:16.576 DEBUG:teuthology.orchestra.run.smithi007:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3 2024-03-25T18:14:16.578 INFO:tasks.ceph.osd.3:Started 2024-03-25T18:14:16.578 DEBUG:tasks.ceph_manager:running osd.4 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.4.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.578 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-25T18:14:16.578 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4 2024-03-25T18:14:16.581 INFO:tasks.ceph.osd.4:Started 2024-03-25T18:14:16.581 DEBUG:tasks.ceph_manager:running osd.5 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.5.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.581 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-25T18:14:16.581 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5 2024-03-25T18:14:16.583 INFO:tasks.ceph.osd.5:Started 2024-03-25T18:14:16.583 DEBUG:tasks.ceph_manager:running osd.6 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.6.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.583 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-25T18:14:16.583 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6 2024-03-25T18:14:16.585 INFO:tasks.ceph.osd.6:Started 2024-03-25T18:14:16.585 DEBUG:tasks.ceph_manager:running osd.7 under valgrind with args ['cd', '/home/ubuntu/cephtest', Raw('&&'), 'sudo', 'adjust-ulimits', 'ceph-coverage', '/home/ubuntu/cephtest/archive/coverage', 'daemon-helper', 'term', 'env', 'OPENSSL_ia32cap=~0x1000000000000000', 'valgrind', '--trace-children=no', '--child-silent-after-fork=yes', '--soname-synonyms=somalloc=*tcmalloc*', '--num-callers=50', '--suppressions=/home/ubuntu/cephtest/valgrind.supp', '--xml=yes', '--xml-file=/var/log/ceph/valgrind/osd.7.log', '--time-stamp=yes', '--vgdb=yes', '--exit-on-first-error=yes', '--error-exitcode=42', '--tool=memcheck'] 2024-03-25T18:14:16.585 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-25T18:14:16.586 DEBUG:teuthology.orchestra.run.smithi183:> cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7 2024-03-25T18:14:16.588 INFO:tasks.ceph.osd.7:Started 2024-03-25T18:14:16.588 DEBUG:teuthology.orchestra.run.smithi007:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.754+0000 7f6314e81640 10 monclient: get_monmap_and_config 2024-03-25T18:14:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.754+0000 7f6314e81640 10 monclient: build_initial_monmap 2024-03-25T18:14:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.754+0000 7f6314e81640 10 monclient: monmap: 2024-03-25T18:14:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-25T18:14:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:16.755 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:14:16.755171+0000 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:14:16.755171+0000 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.754+0000 7f6314e81640 1 Processor -- start 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.754+0000 7f6314e81640 1 -- start start 2024-03-25T18:14:16.756 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.754+0000 7f6314e81640 10 monclient: init 2024-03-25T18:14:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 10 monclient(hunting): picked mon.noname-a con 0x7f63100b4b50 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 10 monclient(hunting): picked mon.noname-b con 0x7f631013ff20 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 10 monclient(hunting): picked mon.noname-c con 0x7f6310142dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:16.757 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63100ab440 con 0x7f63100b4b50 2024-03-25T18:14:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63100ab6b0 con 0x7f6310142dd0 2024-03-25T18:14:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63100a9f10 con 0x7f631013ff20 2024-03-25T18:14:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.755+0000 7f6314e81640 10 monclient(hunting): authenticate will time out at 7298.860558s 2024-03-25T18:14:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.756+0000 7f630e59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f63100b4b50 0x7f631013f810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54150/0 (socket says 172.21.15.7:54150) 2024-03-25T18:14:16.758 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.756+0000 7f630e59c640 1 -- 172.21.15.7:0/892893436 learned_addr learned my addr 172.21.15.7:0/892893436 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:16.882 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.882+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970893334 0 0) 0x7f63100a9f10 con 0x7f631013ff20 2024-03-25T18:14:16.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.882+0000 7f630d59a640 10 monclient(hunting): _negotiate 2024-03-25T18:14:16.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.882+0000 7f630d59a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:16.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.882+0000 7f630d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:16.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.882+0000 7f630d59a640 10 monclient(hunting): my global_id is 4172 2024-03-25T18:14:16.883 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.882+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f8003520 con 0x7f631013ff20 2024-03-25T18:14:16.890 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3906407411 0 0) 0x7f62f8003520 con 0x7f631013ff20 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 >> v1:172.21.15.183:6789/0 conn(0x7f6310142dd0 legacy=0x7f631014df50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 >> v1:172.21.15.7:6789/0 conn(0x7f63100b4b50 legacy=0x7f631013f810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 10 monclient: _finish_hunting 0 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 1 monclient: found mon.noname-b 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63100aa0a0 con 0x7f631013ff20 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 10 monclient: _finish_auth 0 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 10 monclient: _check_auth_tickets 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.889+0000 7f630d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.890+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6304001690 con 0x7f631013ff20 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.890+0000 7f6314e81640 5 monclient: authenticate success, global_id 4172 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.890+0000 7f6314e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:16.891 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.890+0000 7f630d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.890+0000 7f630d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.890+0000 7f630d59a640 10 monclient: dump: 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:16.892 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.890+0000 7f6314e81640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63040017f0 con 0x7f631013ff20 2024-03-25T18:14:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f630d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f630d59a640 1 -- 172.21.15.7:0/892893436 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6304001b80 con 0x7f631013ff20 2024-03-25T18:14:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f630d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f630d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f630d59a640 10 monclient: dump: 2024-03-25T18:14:16.907 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f6314e81640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:16.908 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.906+0000 7f6314e81640 4 set_mon_vals no callback set 2024-03-25T18:14:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.907+0000 7f6314e81640 10 monclient: shutdown 2024-03-25T18:14:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.907+0000 7f6314e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.907+0000 7f6314e81640 1 -- 172.21.15.7:0/892893436 >> v1:172.21.15.7:6790/0 conn(0x7f631013ff20 legacy=0x7f63101426c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.908+0000 7f6314e81640 1 -- 172.21.15.7:0/892893436 shutdown_connections 2024-03-25T18:14:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.908+0000 7f6314e81640 1 -- 172.21.15.7:0/892893436 >> 172.21.15.7:0/892893436 conn(0x7f6310066920 msgr2=0x7f6310066d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.908+0000 7f6314e81640 1 -- 172.21.15.7:0/892893436 shutdown_connections 2024-03-25T18:14:16.909 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.908+0000 7f6314e81640 1 -- 172.21.15.7:0/892893436 wait complete. 2024-03-25T18:14:16.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.909+0000 7f6314e81640 10 monclient: build_initial_monmap 2024-03-25T18:14:16.910 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.909+0000 7f6314e81640 10 monclient: monmap: 2024-03-25T18:14:16.910 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 0 2024-03-25T18:14:16.910 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:created 0.000000 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.909+0000 7f6314e81640 1 librados: starting msgr at 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.909+0000 7f6314e81640 1 librados: starting objecter 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.909+0000 7f6314e81640 1 Processor -- start 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.910+0000 7f6314e81640 1 -- start start 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.910+0000 7f6314e81640 1 librados: setting wanted keys 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.910+0000 7f6314e81640 1 librados: calling monclient init 2024-03-25T18:14:16.911 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.910+0000 7f6314e81640 10 monclient: init 2024-03-25T18:14:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.911+0000 7f6314e81640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.911+0000 7f6314e81640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.911+0000 7f6314e81640 10 monclient(hunting): picked mon.noname-a con 0x7f63100b4b50 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.911+0000 7f6314e81640 10 monclient(hunting): picked mon.noname-c con 0x7f631013ff20 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f6314e81640 10 monclient(hunting): picked mon.noname-b con 0x7f6310142dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:16.913 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f630e59c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f63100b4b50 0x7f63101dabd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54154/0 (socket says 172.21.15.7:54154) 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f630e59c640 1 -- 172.21.15.7:0/1875842325 learned_addr learned my addr 172.21.15.7:0/1875842325 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f631013a5b0 con 0x7f63100b4b50 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6310144420 con 0x7f6310142dd0 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6310144690 con 0x7f631013ff20 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f62feffd640 10 client.?.objecter ms_handle_connect 0x7f63100b4b50 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f62feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f62feffd640 10 client.?.objecter ms_handle_connect 0x7f631013ff20 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f62feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f6314e81640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:16.914 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.912+0000 7f6314e81640 10 monclient(hunting): authenticate will time out at 7299.017579s 2024-03-25T18:14:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.916+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4002936538 0 0) 0x7f6310144690 con 0x7f631013ff20 2024-03-25T18:14:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.916+0000 7f62feffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.916+0000 7f62feffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.916+0000 7f62feffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.916+0000 7f62feffd640 10 monclient(hunting): my global_id is 4175 2024-03-25T18:14:16.917 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.916+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62f0004eb0 con 0x7f631013ff20 2024-03-25T18:14:16.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.924+0000 7f62feffd640 10 client.?.objecter ms_handle_connect 0x7f6310142dd0 2024-03-25T18:14:16.925 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.924+0000 7f62feffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.929+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3827291163 0 0) 0x7f62f0004eb0 con 0x7f631013ff20 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.929+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6310144690 con 0x7f631013ff20 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.929+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6304001b00 con 0x7f631013ff20 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.929+0000 7f62feffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.929+0000 7f62feffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.929+0000 7f62feffd640 10 monclient(hunting): dump: 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:16.930 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:16.931 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:16.931 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:16.931 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:16.931 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:16.931 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:16.929+0000 7f62feffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2192720744 0 0) 0x7f6310144690 con 0x7f631013ff20 2024-03-25T18:14:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 >> v1:172.21.15.183:6789/0 conn(0x7f6310142dd0 legacy=0x7f63101e2f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 >> v1:172.21.15.7:6789/0 conn(0x7f63100b4b50 legacy=0x7f63101dabd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 1 monclient: found mon.c 2024-03-25T18:14:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:17.014 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f631013a140 con 0x7f631013ff20 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f62feffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 5 monclient: authenticate success, global_id 4175 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 10 monclient: _renew_subs 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f631013aa40 con 0x7f631013ff20 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 10 client.4175.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 10 monclient: _renew_subs 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f63101db4c0 con 0x7f631013ff20 2024-03-25T18:14:17.015 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.013+0000 7f6314e81640 1 librados: init done 2024-03-25T18:14:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.014+0000 7f6314e81640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.014+0000 7f6314e81640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.014+0000 7f6314e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:17.016 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.014+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f631004f730 con 0x7f631013ff20 2024-03-25T18:14:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.138+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f63040038a0 con 0x7f631013ff20 2024-03-25T18:14:17.139 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.138+0000 7f62feffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.138+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6304003bb0 con 0x7f631013ff20 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.138+0000 7f62feffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.138+0000 7f62feffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.138+0000 7f62feffd640 10 monclient: dump: 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:epoch 1 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:14:17.140 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.138+0000 7f630cd99640 4 set_mon_vals no callback set 2024-03-25T18:14:17.141 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.139+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 2) v1 ==== 44919+0+0 (unknown 3100637744 0 0) 0x7f630400ec90 con 0x7f631013ff20 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f63040146d0 con 0x7f631013ff20 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 10 client.4175.objecter ms_dispatch 0x7f6310000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 3 client.4175.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 3 client.4175.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 20 client.4175.objecter dump_active .. 0 homeless 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f6304013990 con 0x7f631013ff20 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:17.151 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.150+0000 7f62feffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:17.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.253+0000 7f6314e81640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.253+0000 7f6314e81640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.253+0000 7f6314e81640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:17.254 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.253+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f63100681c0 con 0x7f631013ff20 2024-03-25T18:14:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.344+0000 7f62feffd640 1 -- 172.21.15.7:0/1875842325 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f6304012170 con 0x7f631013ff20 2024-03-25T18:14:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.344+0000 7f62feffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:17.345 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.344+0000 7f62feffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:17.345 INFO:teuthology.orchestra.run.smithi007.stdout: 2024-03-25T18:14:17.345 INFO:teuthology.orchestra.run.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 10 librados: watch_flush enter 2024-03-25T18:14:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 10 librados: watch_flush exit 2024-03-25T18:14:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 20 client.4175.objecter shutdown clearing up homeless session... 2024-03-25T18:14:17.346 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 10 client.4175.objecter successfully canceled tick 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 10 monclient: shutdown 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 >> v1:172.21.15.7:6790/0 conn(0x7f631013ff20 legacy=0x7f6310145e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 shutdown_connections 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 >> 172.21.15.7:0/1875842325 conn(0x7f6310066920 msgr2=0x7f6310066d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 shutdown_connections 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 1 -- 172.21.15.7:0/1875842325 wait complete. 2024-03-25T18:14:17.347 INFO:teuthology.orchestra.run.smithi007.stderr:2024-03-25T18:14:17.345+0000 7f6314e81640 1 librados: shutdown 2024-03-25T18:14:17.358 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-25T18:14:17.358 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-25T18:14:17.857 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:17.856+0000 7f3bb8b51640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-25T18:14:18.004 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:14:18.003+0000 7f3bb8b51640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-25T18:14:18.163 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.310+0000 7f10385a8640 10 monclient: get_monmap_and_config 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.310+0000 7f10385a8640 10 monclient: build_initial_monmap 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.310+0000 7f10385a8640 10 monclient: monmap: 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:14:18.311185+0000 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:14:18.311185+0000 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:18.311 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:18.312 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:18.312 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:18.312 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.311+0000 7f10385a8640 1 Processor -- start 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.311+0000 7f10385a8640 1 -- start start 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.311+0000 7f10385a8640 10 monclient: init 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.312+0000 7f10385a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.312+0000 7f10385a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.312+0000 7f10385a8640 10 monclient(hunting): picked mon.noname-c con 0x7f10300b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.312+0000 7f10385a8640 10 monclient(hunting): picked mon.noname-b con 0x7f10300b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.312+0000 7f10385a8640 10 monclient(hunting): picked mon.noname-a con 0x7f103013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.312+0000 7f10385a8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10300ab0e0 con 0x7f103013c480 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.313+0000 7f10385a8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10300ab3c0 con 0x7f10300b4ff0 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.313+0000 7f10385a8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10300a9f30 con 0x7f10300b4670 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.313+0000 7f10385a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:18.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.313+0000 7f10385a8640 10 monclient(hunting): authenticate will time out at 7300.417713s 2024-03-25T18:14:18.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.313+0000 7f1035b43640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f10300b4670 0x7f10301425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:44214/0 (socket says 172.21.15.7:44214) 2024-03-25T18:14:18.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.313+0000 7f1035b43640 1 -- 172.21.15.7:0/3620860860 learned_addr learned my addr 172.21.15.7:0/3620860860 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:18.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.316+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3082173503 0 0) 0x7f10300a9f30 con 0x7f10300b4670 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.316+0000 7f1035342640 10 monclient(hunting): _negotiate 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.316+0000 7f1035342640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.316+0000 7f1035342640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.316+0000 7f1035342640 10 monclient(hunting): my global_id is 4178 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.316+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f101c003520 con 0x7f10300b4670 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2218122099 0 0) 0x7f10300ab0e0 con 0x7f103013c480 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): _negotiate 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): my global_id is 4173 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10300a9f30 con 0x7f103013c480 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547075261 0 0) 0x7f10300ab3c0 con 0x7f10300b4ff0 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): _negotiate 2024-03-25T18:14:18.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:18.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:18.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 10 monclient(hunting): my global_id is 4165 2024-03-25T18:14:18.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.317+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f10300ab0e0 con 0x7f10300b4ff0 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2407803917 0 0) 0x7f101c003520 con 0x7f10300b4670 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 >> v1:172.21.15.183:6789/0 conn(0x7f10300b4ff0 legacy=0x7f103013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 >> v1:172.21.15.7:6789/0 conn(0x7f103013c480 legacy=0x7f103014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 10 monclient: _finish_hunting 0 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 1 monclient: found mon.noname-b 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10300a9b70 con 0x7f10300b4670 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 10 monclient: _finish_auth 0 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 10 monclient: _check_auth_tickets 2024-03-25T18:14:18.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1020001690 con 0x7f10300b4670 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f10385a8640 5 monclient: authenticate success, global_id 4178 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f10385a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f1035342640 10 monclient: dump: 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:18.325 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:18.326 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:18.326 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:18.326 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:18.326 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.322+0000 7f10385a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:18.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.324+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f10200017f0 con 0x7f10300b4670 2024-03-25T18:14:18.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.324+0000 7f1035342640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:18.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.324+0000 7f10385a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.324+0000 7f10385a8640 4 set_mon_vals no callback set 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.324+0000 7f1035342640 1 -- 172.21.15.7:0/3620860860 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1020001c00 con 0x7f10300b4670 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.324+0000 7f1035342640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f1035342640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f1035342640 10 monclient: dump: 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:18.329 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f10385a8640 10 monclient: shutdown 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f10385a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f10385a8640 1 -- 172.21.15.7:0/3620860860 >> v1:172.21.15.7:6790/0 conn(0x7f10300b4670 legacy=0x7f10301425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f10385a8640 1 -- 172.21.15.7:0/3620860860 shutdown_connections 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f10385a8640 1 -- 172.21.15.7:0/3620860860 >> 172.21.15.7:0/3620860860 conn(0x7f10300617a0 msgr2=0x7f1030061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f10385a8640 1 -- 172.21.15.7:0/3620860860 shutdown_connections 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.325+0000 7f10385a8640 1 -- 172.21.15.7:0/3620860860 wait complete. 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.326+0000 7f10385a8640 10 monclient: build_initial_monmap 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.326+0000 7f10385a8640 10 monclient: monmap: 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:18.330 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.327+0000 7f10385a8640 1 librados: starting msgr at 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.327+0000 7f10385a8640 1 librados: starting objecter 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.327+0000 7f10385a8640 1 Processor -- start 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.327+0000 7f10385a8640 1 -- start start 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.327+0000 7f10385a8640 1 librados: setting wanted keys 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.327+0000 7f10385a8640 1 librados: calling monclient init 2024-03-25T18:14:18.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.327+0000 7f10385a8640 10 monclient: init 2024-03-25T18:14:18.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.328+0000 7f10385a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:18.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.328+0000 7f10385a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:18.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.328+0000 7f10385a8640 10 monclient(hunting): picked mon.noname-c con 0x7f10300b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:18.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f10385a8640 10 monclient(hunting): picked mon.noname-a con 0x7f10300b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:18.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f10385a8640 10 monclient(hunting): picked mon.noname-b con 0x7f103013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f10385a8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f103013ef50 con 0x7f10300b4ff0 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f10385a8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1030144af0 con 0x7f103013c480 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f1036344640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f10300b4670 0x7f10301457d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:44222/0 (socket says 172.21.15.7:44222) 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f1036344640 1 -- 172.21.15.7:0/3171552320 learned_addr learned my addr 172.21.15.7:0/3171552320 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1030144d60 con 0x7f10300b4670 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f10385a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f1026ffd640 10 client.?.objecter ms_handle_connect 0x7f10300b4670 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f1026ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.329+0000 7f10385a8640 10 monclient(hunting): authenticate will time out at 7300.434517s 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.330+0000 7f1026ffd640 10 client.?.objecter ms_handle_connect 0x7f10300b4ff0 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.330+0000 7f1026ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.330+0000 7f1026ffd640 10 client.?.objecter ms_handle_connect 0x7f103013c480 2024-03-25T18:14:18.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.330+0000 7f1026ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:18.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 276462758 0 0) 0x7f1030144d60 con 0x7f10300b4670 2024-03-25T18:14:18.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:18.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:18.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:18.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): my global_id is 4181 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1014004eb0 con 0x7f10300b4670 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3724486789 0 0) 0x7f103013ef50 con 0x7f10300b4ff0 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.333+0000 7f1026ffd640 10 monclient(hunting): my global_id is 4176 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.334+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1030144d60 con 0x7f10300b4ff0 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.334+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497909016 0 0) 0x7f1030144af0 con 0x7f103013c480 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.334+0000 7f1026ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.334+0000 7f1026ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.334+0000 7f1026ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.334+0000 7f1026ffd640 10 monclient(hunting): my global_id is 4168 2024-03-25T18:14:18.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.334+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f103013ef50 con 0x7f103013c480 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3200502009 0 0) 0x7f1014004eb0 con 0x7f10300b4670 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1030144af0 con 0x7f10300b4670 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f102c001a30 con 0x7f10300b4670 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:18.343 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.341+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3374465180 0 0) 0x7f1030144d60 con 0x7f10300b4ff0 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.342+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1014004eb0 con 0x7f10300b4ff0 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.342+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1020001b00 con 0x7f10300b4ff0 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.342+0000 7f1026ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.342+0000 7f1026ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:18.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.342+0000 7f1026ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.342+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2682517684 0 0) 0x7f103013ef50 con 0x7f103013c480 2024-03-25T18:14:18.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.342+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1030144d60 con 0x7f103013c480 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.344+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1028001b70 con 0x7f103013c480 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.344+0000 7f1026ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.344+0000 7f1026ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.344+0000 7f1026ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.346 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:18.347 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:18.347 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:18.347 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:18.347 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:18.347 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1461535473 0 0) 0x7f1030144af0 con 0x7f10300b4670 2024-03-25T18:14:18.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 >> v1:172.21.15.183:6789/0 conn(0x7f103013c480 legacy=0x7f10301e2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 >> v1:172.21.15.7:6789/0 conn(0x7f10300b4ff0 legacy=0x7f1030145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:18.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:18.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 1 monclient: found mon.c 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1030145040 con 0x7f10300b4670 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.347+0000 7f1026ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f1026ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f1026ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 5 monclient: authenticate success, global_id 4181 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 10 monclient: _renew_subs 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1030143660 con 0x7f10300b4670 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 10 monclient: _renew_subs 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:18.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f103013dd60 con 0x7f10300b4670 2024-03-25T18:14:18.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.348+0000 7f10385a8640 1 librados: init done 2024-03-25T18:14:18.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.349+0000 7f10385a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:18.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.349+0000 7f10385a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:18.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.349+0000 7f10385a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:18.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.349+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1030143850 con 0x7f10300b4670 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.378+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f102c001b90 con 0x7f10300b4670 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.378+0000 7f1026ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.378+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f102c0038c0 con 0x7f10300b4670 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.378+0000 7f1026ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.378+0000 7f1026ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.378+0000 7f1026ffd640 10 monclient: dump: 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:18.380 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.378+0000 7f1034b41640 4 set_mon_vals no callback set 2024-03-25T18:14:18.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.379+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 3) v1 ==== 89591+0+0 (unknown 711513813 0 0) 0x7f102c003ae0 con 0x7f10300b4670 2024-03-25T18:14:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.380+0000 7f1035b43640 10 monclient: get_auth_request con 0x7f101406df90 auth_method 0 2024-03-25T18:14:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.380+0000 7f1026ffd640 10 client.4181.objecter ms_handle_connect 0x7f101406df90 2024-03-25T18:14:18.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.383+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f102c054640 con 0x7f10300b4670 2024-03-25T18:14:18.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.383+0000 7f1026ffd640 10 client.4181.objecter ms_dispatch 0x7f1030000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:18.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.383+0000 7f1026ffd640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:18.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.383+0000 7f1026ffd640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:18.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.383+0000 7f1026ffd640 20 client.4181.objecter dump_active .. 0 homeless 2024-03-25T18:14:18.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.819+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f102c050150 con 0x7f10300b4670 2024-03-25T18:14:18.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.819+0000 7f1026ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:18.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:18.819+0000 7f1026ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:19.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.123+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f102c04fc00 con 0x7f10300b4670 2024-03-25T18:14:19.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.187+0000 7f10385a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:19.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.188+0000 7f10385a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:19.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.188+0000 7f10385a8640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:19.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.188+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f10301437f0 con 0x7f10300b4670 2024-03-25T18:14:19.215 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.213+0000 7f1026ffd640 1 -- 172.21.15.7:0/3171552320 <== mon.2 v1:172.21.15.7:6790/0 11 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f102c001890 con 0x7f10300b4670 2024-03-25T18:14:19.215 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.213+0000 7f1026ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:19.215 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.213+0000 7f1026ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:19.215 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:14:19.215 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:19.218 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 10 librados: watch_flush enter 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 10 librados: watch_flush exit 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 20 client.4181.objecter shutdown clearing up homeless session... 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 10 client.4181.objecter successfully canceled tick 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 >> v1:172.21.15.7:6800/1717564767 conn(0x7f101406df90 legacy=0x7f1014070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 10 monclient: shutdown 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 >> v1:172.21.15.7:6790/0 conn(0x7f10300b4670 legacy=0x7f10301457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 shutdown_connections 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 >> 172.21.15.7:0/3171552320 conn(0x7f10300617a0 msgr2=0x7f1030140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 shutdown_connections 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.216+0000 7f10385a8640 1 -- 172.21.15.7:0/3171552320 wait complete. 2024-03-25T18:14:19.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:19.217+0000 7f10385a8640 1 librados: shutdown 2024-03-25T18:14:19.235 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:23.356 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:06.702 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.364 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:06.713 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.416 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:06.763 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.419 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:06.754 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.426 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:06.775 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.495 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:06.830 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.760 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:07.110 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.820 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:07.170 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.846 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:07.185 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.909 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:07.249 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:23.969 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-25T18:14:23.963+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:23.975 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-25T18:14:23.968+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:24.039 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-25T18:14:24.033+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:24.147 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:07.486 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.207 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:07.546 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.209 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:07.544 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.270 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:07.605 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.360 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:07.693 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.403 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-25T18:14:24.396+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:24.434 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:07.767 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.439 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:07.786 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.450 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:07.799 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.478 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-25T18:14:24.470+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:24.519 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:07.854 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.805 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-25T18:14:24.793+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:24.808 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:08.155 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.821 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:08.170 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.833 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-25T18:14:24.826+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:24.858 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:08.209 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.887 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:08.222 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:24.968 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:08.308 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.016 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-25T18:14:25.009+0000 5dc3480 -1 Falling back to public interface 2024-03-25T18:14:25.136 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:08.486 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.143 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:08.490 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.188 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:08.535 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.225 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:08.576 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.245 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:08.580 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.264 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.603 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.304 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:08.639 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.358 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:08.697 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.400 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:08.749 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.477 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:08.824 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.478 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:08.812 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.539 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:08.874 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.545 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:08.896 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.639 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:08.979 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.672 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:09.021 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.676 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.016 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.683 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.018 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.764 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:09.111 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.824 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.174 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.835 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:09.170 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.853 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.187 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.936 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:09.286 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.947 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.287 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:25.965 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.305 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.041 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:26.042 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.377 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.053 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:09.400 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.091 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.441 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.127 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:09.462 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.130+0000 7f74ea946640 10 monclient: get_monmap_and_config 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.130+0000 7f74ea946640 10 monclient: build_initial_monmap 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.130+0000 7f74ea946640 10 monclient: monmap: 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:14:26.131568+0000 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:14:26.131568+0000 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:26.131 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 1 Processor -- start 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 1 -- start start 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient: init 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient(hunting): picked mon.noname-a con 0x7f74e406c3e0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient(hunting): picked mon.noname-b con 0x7f74e40b0e00 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient(hunting): picked mon.noname-c con 0x7f74e41465b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e41469d0 con 0x7f74e406c3e0 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e40adfc0 con 0x7f74e41465b0 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e40ae2a0 con 0x7f74e40b0e00 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.131+0000 7f74ea946640 10 monclient(hunting): authenticate will time out at 7308.236541s 2024-03-25T18:14:26.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.132+0000 7f74e3fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f74e406c3e0 0x7f74e40b06f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:60890/0 (socket says 172.21.15.7:60890) 2024-03-25T18:14:26.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.132+0000 7f74e3fff640 1 -- 172.21.15.7:0/2415367166 learned_addr learned my addr 172.21.15.7:0/2415367166 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:26.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.134+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2566488757 0 0) 0x7f74e40adfc0 con 0x7f74e41465b0 2024-03-25T18:14:26.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.134+0000 7f74e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:26.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.134+0000 7f74e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:26.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.134+0000 7f74e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:26.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.134+0000 7f74e2ffd640 10 monclient(hunting): my global_id is 4195 2024-03-25T18:14:26.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.134+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74d0003520 con 0x7f74e41465b0 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3866580494 0 0) 0x7f74e41469d0 con 0x7f74e406c3e0 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): my global_id is 4203 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e40adfc0 con 0x7f74e406c3e0 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2061334567 0 0) 0x7f74e40ae2a0 con 0x7f74e40b0e00 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 10 monclient(hunting): my global_id is 4208 2024-03-25T18:14:26.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.135+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e41469d0 con 0x7f74e40b0e00 2024-03-25T18:14:26.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 609388023 0 0) 0x7f74d0003520 con 0x7f74e41465b0 2024-03-25T18:14:26.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 >> v1:172.21.15.7:6790/0 conn(0x7f74e40b0e00 legacy=0x7f74e4145ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 >> v1:172.21.15.7:6789/0 conn(0x7f74e406c3e0 legacy=0x7f74e40b06f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e406d210 con 0x7f74e41465b0 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f74dc001a10 con 0x7f74e41465b0 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74ea946640 5 monclient: authenticate success, global_id 4195 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74ea946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:26.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74e2ffd640 10 monclient: dump: 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:26.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.137+0000 7f74ea946640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74dc001b70 con 0x7f74e41465b0 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74e2ffd640 1 -- 172.21.15.7:0/2415367166 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f74dc001f00 con 0x7f74e41465b0 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74ea946640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74ea946640 4 set_mon_vals no callback set 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74e2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.139+0000 7f74e2ffd640 10 monclient: dump: 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.142 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 10 monclient: shutdown 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 1 -- 172.21.15.7:0/2415367166 >> v1:172.21.15.183:6789/0 conn(0x7f74e41465b0 legacy=0x7f74e414ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 1 -- 172.21.15.7:0/2415367166 shutdown_connections 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 1 -- 172.21.15.7:0/2415367166 >> 172.21.15.7:0/2415367166 conn(0x7f74e40b5cf0 msgr2=0x7f74e40b6100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 1 -- 172.21.15.7:0/2415367166 shutdown_connections 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 1 -- 172.21.15.7:0/2415367166 wait complete. 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 10 monclient: build_initial_monmap 2024-03-25T18:14:26.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.140+0000 7f74ea946640 10 monclient: monmap: 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.141+0000 7f74ea946640 1 librados: starting msgr at 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.141+0000 7f74ea946640 1 librados: starting objecter 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.141+0000 7f74ea946640 1 Processor -- start 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.141+0000 7f74ea946640 1 -- start start 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.141+0000 7f74ea946640 1 librados: setting wanted keys 2024-03-25T18:14:26.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.141+0000 7f74ea946640 1 librados: calling monclient init 2024-03-25T18:14:26.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.141+0000 7f74ea946640 10 monclient: init 2024-03-25T18:14:26.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:26.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 10 monclient(hunting): picked mon.noname-b con 0x7f74e406c3e0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 10 monclient(hunting): picked mon.noname-c con 0x7f74e40b0e00 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 10 monclient(hunting): picked mon.noname-a con 0x7f74e41465b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e411d970 con 0x7f74e41465b0 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e411f130 con 0x7f74e406c3e0 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74e411e5a0 con 0x7f74e40b0e00 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74ea946640 10 monclient(hunting): authenticate will time out at 7308.246841s 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e37fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f74e40b0e00 0x7f74e41db190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:46410/0 (socket says 172.21.15.7:46410) 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e37fe640 1 -- 172.21.15.7:0/2423824179 learned_addr learned my addr 172.21.15.7:0/2423824179 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e0ff9640 10 client.?.objecter ms_handle_connect 0x7f74e40b0e00 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e0ff9640 10 client.?.objecter ms_handle_connect 0x7f74e41465b0 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e0ff9640 10 client.?.objecter ms_handle_connect 0x7f74e406c3e0 2024-03-25T18:14:26.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.142+0000 7f74e0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.144+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054296566 0 0) 0x7f74e411f130 con 0x7f74e406c3e0 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.144+0000 7f74e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.144+0000 7f74e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.144+0000 7f74e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.144+0000 7f74e0ff9640 10 monclient(hunting): my global_id is 4198 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.144+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74c4004eb0 con 0x7f74e406c3e0 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.145+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3870150021 0 0) 0x7f74e411d970 con 0x7f74e41465b0 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.145+0000 7f74e0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.145+0000 7f74e0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.145+0000 7f74e0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.145+0000 7f74e0ff9640 10 monclient(hunting): my global_id is 4206 2024-03-25T18:14:26.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.145+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74e411f130 con 0x7f74e41465b0 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3126819355 0 0) 0x7f74e411f130 con 0x7f74e41465b0 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74e411d970 con 0x7f74e41465b0 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f74dc001e80 con 0x7f74e41465b0 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:26.150 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 923376571 0 0) 0x7f74c4004eb0 con 0x7f74e406c3e0 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f74e411f130 con 0x7f74e406c3e0 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f74d4001970 con 0x7f74e406c3e0 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.149+0000 7f74e0ff9640 10 monclient(hunting): dump: 2024-03-25T18:14:26.151 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:26.152 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:26.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 311910532 0 0) 0x7f74e411f130 con 0x7f74e406c3e0 2024-03-25T18:14:26.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 >> v1:172.21.15.7:6790/0 conn(0x7f74e40b0e00 legacy=0x7f74e41db190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 >> v1:172.21.15.7:6789/0 conn(0x7f74e41465b0 legacy=0x7f74e413e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 1 monclient: found mon.b 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74e411e730 con 0x7f74e406c3e0 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74e0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 5 monclient: authenticate success, global_id 4198 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 10 monclient: _renew_subs 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:26.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74e41db8a0 con 0x7f74e406c3e0 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 10 client.4198.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 10 monclient: _renew_subs 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f74e41445e0 con 0x7f74e406c3e0 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.152+0000 7f74ea946640 1 librados: init done 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.154+0000 7f74ea946640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.154+0000 7f74ea946640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.154+0000 7f74ea946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.154+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74e411db20 con 0x7f74e406c3e0 2024-03-25T18:14:26.206 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:09.555 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.211 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.551 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.221 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.555 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.244 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.583 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.331 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.667 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.342 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:09.689 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.362+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f74d40047e0 con 0x7f74e406c3e0 2024-03-25T18:14:26.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.362+0000 7f74e0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:26.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.362+0000 7f74e1ffb640 4 set_mon_vals no callback set 2024-03-25T18:14:26.368 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.719 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.392+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f74d4004950 con 0x7f74e406c3e0 2024-03-25T18:14:26.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.392+0000 7f74e0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.392+0000 7f74e0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.392+0000 7f74e0ff9640 10 monclient: dump: 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:26.393 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:26.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.392+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f74d401a9a0 con 0x7f74e406c3e0 2024-03-25T18:14:26.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.393+0000 7f74e37fe640 10 monclient: get_auth_request con 0x7f74c406de90 auth_method 0 2024-03-25T18:14:26.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.393+0000 7f74e0ff9640 10 client.4198.objecter ms_handle_connect 0x7f74c406de90 2024-03-25T18:14:26.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.396+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f74d40546c0 con 0x7f74e406c3e0 2024-03-25T18:14:26.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.396+0000 7f74e0ff9640 10 client.4198.objecter ms_dispatch 0x7f74e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:26.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.396+0000 7f74e0ff9640 3 client.4198.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:26.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.396+0000 7f74e0ff9640 3 client.4198.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:26.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.397+0000 7f74e0ff9640 20 client.4198.objecter dump_active .. 0 homeless 2024-03-25T18:14:26.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.397+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f74d4050000 con 0x7f74e406c3e0 2024-03-25T18:14:26.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.397+0000 7f74e0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:26.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.397+0000 7f74e0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:26.422 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:09.757 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.474 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:09.824 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.477 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:09.817 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.514 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:09.853 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.515 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:09.849 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.584+0000 7f74ea946640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:26.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.584+0000 7f74ea946640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:26.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.584+0000 7f74ea946640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:26.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.584+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74e41dbbf0 con 0x7f74e406c3e0 2024-03-25T18:14:26.620 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:09.956 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.632 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:09.979 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.632+0000 7f74e0ff9640 1 -- 172.21.15.7:0/2423824179 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f74d4052100 con 0x7f74e406c3e0 2024-03-25T18:14:26.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.632+0000 7f74e0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:26.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.632+0000 7f74e0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:26.633 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:14:26.633 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.634+0000 7f74ea946640 10 librados: watch_flush enter 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.634+0000 7f74ea946640 10 librados: watch_flush exit 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.634+0000 7f74ea946640 20 client.4198.objecter shutdown clearing up homeless session... 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.634+0000 7f74ea946640 10 client.4198.objecter successfully canceled tick 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 >> v1:172.21.15.7:6800/1717564767 conn(0x7f74c406de90 legacy=0x7f74c4070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 10 monclient: shutdown 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 >> v1:172.21.15.183:6789/0 conn(0x7f74e406c3e0 legacy=0x7f74e411de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:26.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 shutdown_connections 2024-03-25T18:14:26.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 >> 172.21.15.7:0/2423824179 conn(0x7f74e40b5cf0 msgr2=0x7f74e40aee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:26.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 shutdown_connections 2024-03-25T18:14:26.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 1 -- 172.21.15.7:0/2423824179 wait complete. 2024-03-25T18:14:26.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:26.635+0000 7f74ea946640 1 librados: shutdown 2024-03-25T18:14:26.638 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:09.988 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.645 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:26.710 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:10.044 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.745 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.085 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.745 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:10.094 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.785 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.124 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.815 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.149 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.902 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:10.253 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:26.914 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.249 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.011 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:10.346 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.014 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.354 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.053 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.393 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.105 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.438 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.167 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:10.517 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.171 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:10.518 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.206 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.542 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.225 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:10.572 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.229 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:10.578 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.278 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:10.618 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.283 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:10.632 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.318 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.658 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.393 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:10.727 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.495 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:10.831 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.570 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:10.905 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.594 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:10.933 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.640 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:10.975 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.652 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:11.003 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.685 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.018 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.707 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:11.057 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.777 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.117 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.801 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.136 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.833 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:11.172 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:27.999 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.332 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:28.092 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.431 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:28.173 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:11.512 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:28.355 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.691 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:28.423 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:11.759 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:28.533 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.867 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:28.589 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:11.923 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.504 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:15.851 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:32.715 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:16.064 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:33.089 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:16.428 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:33.144 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:16.479 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:33.163 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:16.513 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:33.451 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:33.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.589+0000 7ff6df4d6640 10 monclient: get_monmap_and_config 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.589+0000 7ff6df4d6640 10 monclient: build_initial_monmap 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.589+0000 7ff6df4d6640 10 monclient: monmap: 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:14:33.590222+0000 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:14:33.590222+0000 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:33.590 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.589+0000 7ff6df4d6640 1 Processor -- start 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 1 -- start start 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient: init 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d80b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d80b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d80b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d80a96f0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d80ab0e0 con 0x7ff6d80b44b0 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d8145f70 con 0x7ff6d80b4e70 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.590+0000 7ff6df4d6640 10 monclient(hunting): authenticate will time out at 7315.695358s 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.591+0000 7ff6dd272640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7ff6d80b4e70 0x7ff6d813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:36290/0 (socket says 172.21.15.7:36290) 2024-03-25T18:14:33.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.591+0000 7ff6dd272640 1 -- 172.21.15.7:0/2607217178 learned_addr learned my addr 172.21.15.7:0/2607217178 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 109721453 0 0) 0x7ff6d80a96f0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): my global_id is 4209 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c4003520 con 0x7ff6d80b1b00 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3641021901 0 0) 0x7ff6d80ab0e0 con 0x7ff6d80b44b0 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): my global_id is 4201 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d80a96f0 con 0x7ff6d80b44b0 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963894138 0 0) 0x7ff6d8145f70 con 0x7ff6d80b4e70 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 10 monclient(hunting): my global_id is 4214 2024-03-25T18:14:33.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.594+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d80ab0e0 con 0x7ff6d80b4e70 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3553017891 0 0) 0x7ff6c4003520 con 0x7ff6d80b1b00 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 >> v1:172.21.15.7:6790/0 conn(0x7ff6d80b4e70 legacy=0x7ff6d813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 >> v1:172.21.15.183:6789/0 conn(0x7ff6d80b44b0 legacy=0x7ff6d8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 1 monclient: found mon.noname-a 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d80a9b80 con 0x7ff6d80b1b00 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 10 monclient: _finish_auth 0 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6df4d6640 5 monclient: authenticate success, global_id 4209 2024-03-25T18:14:33.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.597+0000 7ff6df4d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.598+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff6d00019d0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.598+0000 7ff6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.598+0000 7ff6cffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.598+0000 7ff6cffff640 10 monclient: dump: 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.598+0000 7ff6df4d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:33.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d0001c20 con 0x7ff6d80b1b00 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6cffff640 1 -- 172.21.15.7:0/2607217178 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff6d0001fb0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6cffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6cffff640 10 monclient: dump: 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.608 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6df4d6640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.599+0000 7ff6df4d6640 4 set_mon_vals no callback set 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.600+0000 7ff6df4d6640 10 monclient: shutdown 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.600+0000 7ff6df4d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.600+0000 7ff6df4d6640 1 -- 172.21.15.7:0/2607217178 >> v1:172.21.15.7:6789/0 conn(0x7ff6d80b1b00 legacy=0x7ff6d814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.600+0000 7ff6df4d6640 1 -- 172.21.15.7:0/2607217178 shutdown_connections 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.600+0000 7ff6df4d6640 1 -- 172.21.15.7:0/2607217178 >> 172.21.15.7:0/2607217178 conn(0x7ff6d80617a0 msgr2=0x7ff6d8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.600+0000 7ff6df4d6640 1 -- 172.21.15.7:0/2607217178 shutdown_connections 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.600+0000 7ff6df4d6640 1 -- 172.21.15.7:0/2607217178 wait complete. 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.601+0000 7ff6df4d6640 10 monclient: build_initial_monmap 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.601+0000 7ff6df4d6640 10 monclient: monmap: 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:33.609 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.601+0000 7ff6df4d6640 1 librados: starting msgr at 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.601+0000 7ff6df4d6640 1 librados: starting objecter 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 1 Processor -- start 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 1 -- start start 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 1 librados: setting wanted keys 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 1 librados: calling monclient init 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 10 monclient: init 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 10 monclient(hunting): picked mon.noname-a con 0x7ff6d80b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:33.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 10 monclient(hunting): picked mon.noname-b con 0x7ff6d80b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 10 monclient(hunting): picked mon.noname-c con 0x7ff6d80b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.602+0000 7ff6df4d6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d813c600 con 0x7ff6d80b1b00 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6df4d6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d813c8e0 con 0x7ff6d80b44b0 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6df4d6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d8133860 con 0x7ff6d80b4e70 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6df4d6640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6df4d6640 10 monclient(hunting): authenticate will time out at 7315.707801s 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6dd272640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff6d80b1b00 0x7ff6d8144d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38970/0 (socket says 172.21.15.7:38970) 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6dd272640 1 -- 172.21.15.7:0/3228648800 learned_addr learned my addr 172.21.15.7:0/3228648800 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6cdffb640 10 client.?.objecter ms_handle_connect 0x7ff6d80b1b00 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.603+0000 7ff6cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.604+0000 7ff6cdffb640 10 client.?.objecter ms_handle_connect 0x7ff6d80b44b0 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.604+0000 7ff6cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.604+0000 7ff6cdffb640 10 client.?.objecter ms_handle_connect 0x7ff6d80b4e70 2024-03-25T18:14:33.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.604+0000 7ff6cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249903066 0 0) 0x7ff6d813c600 con 0x7ff6d80b1b00 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): my global_id is 4212 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6bc004eb0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2366798091 0 0) 0x7ff6d813c8e0 con 0x7ff6d80b44b0 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 10 monclient(hunting): my global_id is 4204 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.606+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d813c600 con 0x7ff6d80b44b0 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.607+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 48809028 0 0) 0x7ff6d8133860 con 0x7ff6d80b4e70 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.607+0000 7ff6cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.607+0000 7ff6cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.607+0000 7ff6cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.607+0000 7ff6cdffb640 10 monclient(hunting): my global_id is 4217 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.607+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6d813c8e0 con 0x7ff6d80b4e70 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.611+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1543962080 0 0) 0x7ff6bc004eb0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.611+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6d8133860 con 0x7ff6d80b1b00 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.611+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff6d4001870 con 0x7ff6d80b1b00 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.611+0000 7ff6cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.611+0000 7ff6cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.611+0000 7ff6cdffb640 10 monclient(hunting): dump: 2024-03-25T18:14:33.615 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.611+0000 7ff6cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.613+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4167094935 0 0) 0x7ff6d813c600 con 0x7ff6d80b44b0 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.613+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6bc004eb0 con 0x7ff6d80b44b0 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.613+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff6c80018b0 con 0x7ff6d80b44b0 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.613+0000 7ff6cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.616 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.613+0000 7ff6cdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.613+0000 7ff6cdffb640 10 monclient(hunting): dump: 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:33.617 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.614+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2991900285 0 0) 0x7ff6d813c8e0 con 0x7ff6d80b4e70 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.614+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6d813c600 con 0x7ff6d80b4e70 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.614+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff6d0001d90 con 0x7ff6d80b4e70 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.614+0000 7ff6cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.614+0000 7ff6cdffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.614+0000 7ff6cdffb640 10 monclient(hunting): dump: 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 249922154 0 0) 0x7ff6d8133860 con 0x7ff6d80b1b00 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 >> v1:172.21.15.7:6790/0 conn(0x7ff6d80b4e70 legacy=0x7ff6d81e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 >> v1:172.21.15.183:6789/0 conn(0x7ff6d80b44b0 legacy=0x7ff6d8140160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 1 monclient: found mon.a 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d813ac20 con 0x7ff6d80b1b00 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.615+0000 7ff6cdffb640 10 monclient: _finish_auth 0 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6cdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 5 monclient: authenticate success, global_id 4212 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 10 monclient: _renew_subs 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d813b300 con 0x7ff6d80b1b00 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 10 client.4212.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 10 monclient: _renew_subs 2024-03-25T18:14:33.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:33.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d813b830 con 0x7ff6d80b1b00 2024-03-25T18:14:33.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.616+0000 7ff6df4d6640 1 librados: init done 2024-03-25T18:14:33.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.617+0000 7ff6df4d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:33.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.617+0000 7ff6df4d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:33.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.617+0000 7ff6df4d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:33.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.617+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d80b6f90 con 0x7ff6d80b1b00 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.888+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d40047e0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.888+0000 7ff6cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.888+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff6d4004bf0 con 0x7ff6d80b1b00 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.888+0000 7ff6cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.888+0000 7ff6cdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.888+0000 7ff6cdffb640 10 monclient: dump: 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:33.889 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:33.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.888+0000 7ff6cf7fe640 4 set_mon_vals no callback set 2024-03-25T18:14:33.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.889+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7ff6d4004e10 con 0x7ff6d80b1b00 2024-03-25T18:14:33.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.890+0000 7ff6dca71640 10 monclient: get_auth_request con 0x7ff6bc06e010 auth_method 0 2024-03-25T18:14:33.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.890+0000 7ff6cdffb640 10 client.4212.objecter ms_handle_connect 0x7ff6bc06e010 2024-03-25T18:14:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7ff6d4054780 con 0x7ff6d80b1b00 2024-03-25T18:14:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 10 client.4212.objecter ms_dispatch 0x7ff6d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 3 client.4212.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 3 client.4212.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 20 client.4212.objecter dump_active .. 0 homeless 2024-03-25T18:14:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6d4050290 con 0x7ff6d80b1b00 2024-03-25T18:14:33.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:33.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:33.897+0000 7ff6cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:33.944 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:17.283 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:34.007 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:17.340 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:34.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.109+0000 7ff6df4d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:34.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.109+0000 7ff6df4d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:34.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.109+0000 7ff6df4d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:34.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.109+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6d80b7550 con 0x7ff6d80b1b00 2024-03-25T18:14:34.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.159+0000 7ff6cdffb640 1 -- 172.21.15.7:0/3228648800 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7ff6d40521c0 con 0x7ff6d80b1b00 2024-03-25T18:14:34.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.159+0000 7ff6cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:34.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.159+0000 7ff6cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:34.160 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:14:34.160 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.161+0000 7ff6df4d6640 10 librados: watch_flush enter 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.161+0000 7ff6df4d6640 10 librados: watch_flush exit 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.161+0000 7ff6df4d6640 20 client.4212.objecter shutdown clearing up homeless session... 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.161+0000 7ff6df4d6640 10 client.4212.objecter successfully canceled tick 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 >> v1:172.21.15.7:6800/1717564767 conn(0x7ff6bc06e010 legacy=0x7ff6bc0704d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 10 monclient: shutdown 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 >> v1:172.21.15.7:6789/0 conn(0x7ff6d80b1b00 legacy=0x7ff6d8144d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:34.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 shutdown_connections 2024-03-25T18:14:34.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 >> 172.21.15.7:0/3228648800 conn(0x7ff6d80617a0 msgr2=0x7ff6d80b6680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:34.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 shutdown_connections 2024-03-25T18:14:34.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 1 -- 172.21.15.7:0/3228648800 wait complete. 2024-03-25T18:14:34.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:34.162+0000 7ff6df4d6640 1 librados: shutdown 2024-03-25T18:14:34.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:34.579 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:17.915 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.239 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:21.586 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.243 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:21.590 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.662 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:21.997 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.666 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:22.001 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.707 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:22.057 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.712 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:22.062 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.775 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:22.124 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:38.779 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:22.128 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:39.097 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:22.437 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:39.101 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:22.441 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:39.705 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:23.044 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:39.711 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:23.051 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:39.860 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:23.193 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:39.864 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:23.197 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:40.556 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:23.891 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:40.560 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:23.895 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:40.982 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.102+0000 7f85561a5640 10 monclient: get_monmap_and_config 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.102+0000 7f85561a5640 10 monclient: build_initial_monmap 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.102+0000 7f85561a5640 10 monclient: monmap: 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:14:41.103527+0000 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:14:41.103527+0000 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:41.103 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:41.104 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:41.104 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.103+0000 7f85561a5640 1 Processor -- start 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.103+0000 7f85561a5640 1 -- start start 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.103+0000 7f85561a5640 10 monclient: init 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.103+0000 7f85561a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.103+0000 7f85561a5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 10 monclient(hunting): picked mon.noname-a con 0x7f85500b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 10 monclient(hunting): picked mon.noname-c con 0x7f85500b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 10 monclient(hunting): picked mon.noname-b con 0x7f85500b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85500a96f0 con 0x7f85500b4e70 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85500ab0e0 con 0x7f85500b44b0 2024-03-25T18:14:41.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8550145f70 con 0x7f85500b1b00 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f85561a5640 10 monclient(hunting): authenticate will time out at 7323.208862s 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f854ffff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f85500b1b00 0x7f855014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:41146/0 (socket says 172.21.15.7:41146) 2024-03-25T18:14:41.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.104+0000 7f854ffff640 1 -- 172.21.15.7:0/1661552627 learned_addr learned my addr 172.21.15.7:0/1661552627 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:41.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4066896623 0 0) 0x7f85500ab0e0 con 0x7f85500b44b0 2024-03-25T18:14:41.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): my global_id is 4207 2024-03-25T18:14:41.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8538003520 con 0x7f85500b44b0 2024-03-25T18:14:41.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578291987 0 0) 0x7f85500a96f0 con 0x7f85500b4e70 2024-03-25T18:14:41.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.107+0000 7f854e7fc640 10 monclient(hunting): my global_id is 4215 2024-03-25T18:14:41.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.108+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85500ab0e0 con 0x7f85500b4e70 2024-03-25T18:14:41.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.108+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009190230 0 0) 0x7f8550145f70 con 0x7f85500b1b00 2024-03-25T18:14:41.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.108+0000 7f854e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.108+0000 7f854e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.108+0000 7f854e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.108+0000 7f854e7fc640 10 monclient(hunting): my global_id is 4220 2024-03-25T18:14:41.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.108+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85500a96f0 con 0x7f85500b1b00 2024-03-25T18:14:41.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1188467997 0 0) 0x7f8538003520 con 0x7f85500b44b0 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 >> v1:172.21.15.7:6790/0 conn(0x7f85500b1b00 legacy=0x7f855014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 >> v1:172.21.15.7:6789/0 conn(0x7f85500b4e70 legacy=0x7f855013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85500a9b80 con 0x7f85500b44b0 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8544001690 con 0x7f85500b44b0 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f854e7fc640 10 monclient: dump: 2024-03-25T18:14:41.112 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f85561a5640 5 monclient: authenticate success, global_id 4207 2024-03-25T18:14:41.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.110+0000 7f85561a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:41.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.112+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f85440018b0 con 0x7f85500b44b0 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.112+0000 7f854e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.112+0000 7f854e7fc640 1 -- 172.21.15.7:0/1661552627 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8544001c40 con 0x7f85500b44b0 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.112+0000 7f85561a5640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.112+0000 7f85561a5640 4 set_mon_vals no callback set 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f854e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f854e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f854e7fc640 10 monclient: dump: 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:41.116 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f85561a5640 10 monclient: shutdown 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f85561a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f85561a5640 1 -- 172.21.15.7:0/1661552627 >> v1:172.21.15.183:6789/0 conn(0x7f85500b44b0 legacy=0x7f8550145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f85561a5640 1 -- 172.21.15.7:0/1661552627 shutdown_connections 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f85561a5640 1 -- 172.21.15.7:0/1661552627 >> 172.21.15.7:0/1661552627 conn(0x7f85500617a0 msgr2=0x7f8550061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f85561a5640 1 -- 172.21.15.7:0/1661552627 shutdown_connections 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.113+0000 7f85561a5640 1 -- 172.21.15.7:0/1661552627 wait complete. 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.114+0000 7f85561a5640 10 monclient: build_initial_monmap 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.114+0000 7f85561a5640 10 monclient: monmap: 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:41.117 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.114+0000 7f85561a5640 1 librados: starting msgr at 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.114+0000 7f85561a5640 1 librados: starting objecter 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.115+0000 7f85561a5640 1 Processor -- start 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.115+0000 7f85561a5640 1 -- start start 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.115+0000 7f85561a5640 1 librados: setting wanted keys 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.115+0000 7f85561a5640 1 librados: calling monclient init 2024-03-25T18:14:41.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.115+0000 7f85561a5640 10 monclient: init 2024-03-25T18:14:41.121 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.115+0000 7f85561a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.115+0000 7f85561a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 10 monclient(hunting): picked mon.noname-a con 0x7f85500b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 10 monclient(hunting): picked mon.noname-c con 0x7f85500b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 10 monclient(hunting): picked mon.noname-b con 0x7f85500b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855013bc90 con 0x7f85500b1b00 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855013bf70 con 0x7f85500b4e70 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f855013c250 con 0x7f85500b44b0 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f854f7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f85500b1b00 0x7f8550132d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48102/0 (socket says 172.21.15.7:48102) 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f854f7fe640 1 -- 172.21.15.7:0/3247608698 learned_addr learned my addr 172.21.15.7:0/3247608698 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f85561a5640 10 monclient(hunting): authenticate will time out at 7323.221270s 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f8537fff640 10 client.?.objecter ms_handle_connect 0x7f85500b1b00 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.116+0000 7f8537fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.117+0000 7f8537fff640 10 client.?.objecter ms_handle_connect 0x7f85500b44b0 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.117+0000 7f8537fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.117+0000 7f8537fff640 10 client.?.objecter ms_handle_connect 0x7f85500b4e70 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.117+0000 7f8537fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 622639569 0 0) 0x7f855013bf70 con 0x7f85500b4e70 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): my global_id is 4210 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f853c004eb0 con 0x7f85500b4e70 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3323622765 0 0) 0x7f855013c250 con 0x7f85500b44b0 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 10 monclient(hunting): my global_id is 4223 2024-03-25T18:14:41.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.119+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f855013bf70 con 0x7f85500b44b0 2024-03-25T18:14:41.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.120+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3407751900 0 0) 0x7f855013bc90 con 0x7f85500b1b00 2024-03-25T18:14:41.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.120+0000 7f8537fff640 10 monclient(hunting): _negotiate 2024-03-25T18:14:41.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.120+0000 7f8537fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:41.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.120+0000 7f8537fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:41.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.120+0000 7f8537fff640 10 monclient(hunting): my global_id is 4218 2024-03-25T18:14:41.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.120+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f855013c250 con 0x7f85500b1b00 2024-03-25T18:14:41.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.125+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3216423457 0 0) 0x7f853c004eb0 con 0x7f85500b4e70 2024-03-25T18:14:41.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.125+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f855013bc90 con 0x7f85500b4e70 2024-03-25T18:14:41.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.125+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8548001ba0 con 0x7f85500b4e70 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.125+0000 7f8537fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.125+0000 7f8537fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.125+0000 7f8537fff640 10 monclient(hunting): dump: 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.125+0000 7f8537fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:41.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.127+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3697153922 0 0) 0x7f855013c250 con 0x7f85500b1b00 2024-03-25T18:14:41.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.127+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f853c004eb0 con 0x7f85500b1b00 2024-03-25T18:14:41.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.127+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f85400018d0 con 0x7f85500b1b00 2024-03-25T18:14:41.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.127+0000 7f8537fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.127+0000 7f8537fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.127+0000 7f8537fff640 10 monclient(hunting): dump: 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:41.128 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.128+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 196071882 0 0) 0x7f855013bf70 con 0x7f85500b44b0 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.128+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f855013c250 con 0x7f85500b44b0 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.128+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f85440015d0 con 0x7f85500b44b0 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.128+0000 7f8537fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.128+0000 7f8537fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.128+0000 7f8537fff640 10 monclient(hunting): dump: 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:41.129 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:41.130 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:41.130 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:41.130 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2371139202 0 0) 0x7f855013bc90 con 0x7f85500b4e70 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 >> v1:172.21.15.7:6790/0 conn(0x7f85500b44b0 legacy=0x7f85501e3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 >> v1:172.21.15.7:6789/0 conn(0x7f85500b1b00 legacy=0x7f8550132d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 10 monclient: _finish_hunting 0 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 1 monclient: found mon.b 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8550145d60 con 0x7f85500b4e70 2024-03-25T18:14:41.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 10 monclient: _finish_auth 0 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 10 monclient: _check_auth_tickets 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f8537fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 5 monclient: authenticate success, global_id 4210 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 10 monclient: _renew_subs 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f855013aff0 con 0x7f85500b4e70 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 10 client.4210.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 10 monclient: _renew_subs 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f855014edf0 con 0x7f85500b4e70 2024-03-25T18:14:41.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.130+0000 7f85561a5640 1 librados: init done 2024-03-25T18:14:41.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.131+0000 7f8535ffb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.131+0000 7f8535ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.131+0000 7f8535ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:41.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.131+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8530005080 con 0x7f85500b4e70 2024-03-25T18:14:41.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.485+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8548002010 con 0x7f85500b4e70 2024-03-25T18:14:41.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.485+0000 7f8537fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:41.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.485+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8548004d00 con 0x7f85500b4e70 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.485+0000 7f8537fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.485+0000 7f8537fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.485+0000 7f8537fff640 10 monclient: dump: 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:41.487 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:41.488 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:41.488 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.485+0000 7f854dffb640 4 set_mon_vals no callback set 2024-03-25T18:14:41.488 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.486+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f8548004f20 con 0x7f85500b4e70 2024-03-25T18:14:41.488 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.487+0000 7f854f7fe640 10 monclient: get_auth_request con 0x7f853c06e060 auth_method 0 2024-03-25T18:14:41.488 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.487+0000 7f8537fff640 10 client.4210.objecter ms_handle_connect 0x7f853c06e060 2024-03-25T18:14:41.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.493+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f85480549e0 con 0x7f85500b4e70 2024-03-25T18:14:41.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.493+0000 7f8537fff640 10 client.4210.objecter ms_dispatch 0x7f8550000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:41.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.493+0000 7f8537fff640 3 client.4210.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:41.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.493+0000 7f8537fff640 3 client.4210.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:41.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.493+0000 7f8537fff640 20 client.4210.objecter dump_active .. 0 homeless 2024-03-25T18:14:41.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.494+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f85480504f0 con 0x7f85500b4e70 2024-03-25T18:14:41.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.494+0000 7f8537fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:41.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.494+0000 7f8537fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:41.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.806+0000 7f8535ffb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.806+0000 7f8535ffb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.806+0000 7f8535ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:14:41.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.806+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8530005330 con 0x7f85500b4e70 2024-03-25T18:14:41.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.821+0000 7f8537fff640 1 -- 172.21.15.7:0/3247608698 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f8548052420 con 0x7f85500b4e70 2024-03-25T18:14:41.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.821+0000 7f8537fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:41.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.821+0000 7f8537fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:41.822 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:14:41.822 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 10 librados: watch_flush enter 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 10 librados: watch_flush exit 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 20 client.4210.objecter shutdown clearing up homeless session... 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 10 client.4210.objecter successfully canceled tick 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 >> v1:172.21.15.7:6800/1717564767 conn(0x7f853c06e060 legacy=0x7f853c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 10 monclient: shutdown 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.823+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 >> v1:172.21.15.183:6789/0 conn(0x7f85500b4e70 legacy=0x7f85501e6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:41.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.824+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 shutdown_connections 2024-03-25T18:14:41.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.824+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 >> 172.21.15.7:0/3247608698 conn(0x7f85500617a0 msgr2=0x7f85500b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:41.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.824+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 shutdown_connections 2024-03-25T18:14:41.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.824+0000 7f8535ffb640 1 -- 172.21.15.7:0/3247608698 wait complete. 2024-03-25T18:14:41.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:41.824+0000 7f8535ffb640 1 librados: shutdown 2024-03-25T18:14:41.835 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:42.767 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:00:26.114 35652== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:43.108 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:00:26.443 35653== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:43.189 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:00:26.539 35650== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:43.415 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:00:26.764 35651== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:43.532 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:00:26.871 35536== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:44.173 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:00:27.512 35531== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:44.413 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:00:27.747 35543== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:45.019 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:00:28.354 35542== Warning: unimplemented fcntl command: 1038 2024-03-25T18:14:48.642 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:48.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.763+0000 7fa7a037b640 10 monclient: get_monmap_and_config 2024-03-25T18:14:48.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.763+0000 7fa7a037b640 10 monclient: build_initial_monmap 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.763+0000 7fa7a037b640 10 monclient: monmap: 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:14:48.764083+0000 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:14:48.764083+0000 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:48.764 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:48.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.763+0000 7fa7a037b640 1 Processor -- start 2024-03-25T18:14:48.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.763+0000 7fa7a037b640 1 -- start start 2024-03-25T18:14:48.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.763+0000 7fa7a037b640 10 monclient: init 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 10 monclient(hunting): picked mon.noname-c con 0x7fa7980b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 10 monclient(hunting): picked mon.noname-a con 0x7fa7980b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 10 monclient(hunting): picked mon.noname-b con 0x7fa7980b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7980a96f0 con 0x7fa7980b44b0 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7980ab0e0 con 0x7fa7980b4e70 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa798145f70 con 0x7fa7980b1b00 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.764+0000 7fa7a037b640 10 monclient(hunting): authenticate will time out at 7330.869482s 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.765+0000 7fa79d916640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa7980b44b0 0x7fa798145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48110/0 (socket says 172.21.15.7:48110) 2024-03-25T18:14:48.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.765+0000 7fa79d916640 1 -- 172.21.15.7:0/578180480 learned_addr learned my addr 172.21.15.7:0/578180480 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:48.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.768+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649288322 0 0) 0x7fa7980a96f0 con 0x7fa7980b44b0 2024-03-25T18:14:48.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.768+0000 7fa79d115640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.768+0000 7fa79d115640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.768+0000 7fa79d115640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.768+0000 7fa79d115640 10 monclient(hunting): my global_id is 4221 2024-03-25T18:14:48.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.768+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa784003520 con 0x7fa7980b44b0 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975547711 0 0) 0x7fa798145f70 con 0x7fa7980b1b00 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): my global_id is 4226 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa7980a96f0 con 0x7fa7980b1b00 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4291110382 0 0) 0x7fa7980ab0e0 con 0x7fa7980b4e70 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 10 monclient(hunting): my global_id is 4213 2024-03-25T18:14:48.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.769+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa798145f70 con 0x7fa7980b4e70 2024-03-25T18:14:48.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 363892973 0 0) 0x7fa784003520 con 0x7fa7980b44b0 2024-03-25T18:14:48.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 >> v1:172.21.15.7:6790/0 conn(0x7fa7980b1b00 legacy=0x7fa79814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 >> v1:172.21.15.183:6789/0 conn(0x7fa7980b4e70 legacy=0x7fa79813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 10 monclient: _finish_hunting 0 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 1 monclient: found mon.noname-a 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7980a9b80 con 0x7fa7980b44b0 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 10 monclient: _finish_auth 0 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 10 monclient: _check_auth_tickets 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa788001690 con 0x7fa7980b44b0 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa79d115640 10 monclient: dump: 2024-03-25T18:14:48.774 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa7a037b640 5 monclient: authenticate success, global_id 4221 2024-03-25T18:14:48.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.772+0000 7fa7a037b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:48.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.774+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7880017f0 con 0x7fa7980b44b0 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.774+0000 7fa79d115640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.774+0000 7fa7a037b640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.774+0000 7fa7a037b640 4 set_mon_vals no callback set 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.775+0000 7fa79d115640 1 -- 172.21.15.7:0/578180480 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa788001c00 con 0x7fa7980b44b0 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.775+0000 7fa79d115640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.775+0000 7fa79d115640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.775+0000 7fa79d115640 10 monclient: dump: 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:48.777 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.775+0000 7fa7a037b640 10 monclient: shutdown 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.776+0000 7fa7a037b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.776+0000 7fa7a037b640 1 -- 172.21.15.7:0/578180480 >> v1:172.21.15.7:6789/0 conn(0x7fa7980b44b0 legacy=0x7fa798145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.776+0000 7fa7a037b640 1 -- 172.21.15.7:0/578180480 shutdown_connections 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.776+0000 7fa7a037b640 1 -- 172.21.15.7:0/578180480 >> 172.21.15.7:0/578180480 conn(0x7fa7980617a0 msgr2=0x7fa798061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.776+0000 7fa7a037b640 1 -- 172.21.15.7:0/578180480 shutdown_connections 2024-03-25T18:14:48.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.776+0000 7fa7a037b640 1 -- 172.21.15.7:0/578180480 wait complete. 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.777+0000 7fa7a037b640 10 monclient: build_initial_monmap 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.777+0000 7fa7a037b640 10 monclient: monmap: 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:48.780 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.777+0000 7fa7a037b640 1 librados: starting msgr at 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.777+0000 7fa7a037b640 1 librados: starting objecter 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 1 Processor -- start 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 1 -- start start 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 1 librados: setting wanted keys 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 1 librados: calling monclient init 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 10 monclient: init 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.778+0000 7fa7a037b640 10 monclient(hunting): picked mon.noname-a con 0x7fa7980b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa7a037b640 10 monclient(hunting): picked mon.noname-b con 0x7fa7980b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa7a037b640 10 monclient(hunting): picked mon.noname-c con 0x7fa7980b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:48.781 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa7a037b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa798127fd0 con 0x7fa7980b1b00 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa7a037b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7981dbc80 con 0x7fa7980b44b0 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa7a037b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa79813c3d0 con 0x7fa7980b4e70 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa7a037b640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa7a037b640 10 monclient(hunting): authenticate will time out at 7330.883783s 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa79e117640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa7980b1b00 0x7fa7981dae60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48116/0 (socket says 172.21.15.7:48116) 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa79e117640 1 -- 172.21.15.7:0/3362144258 learned_addr learned my addr 172.21.15.7:0/3362144258 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa78effd640 10 client.?.objecter ms_handle_connect 0x7fa7980b1b00 2024-03-25T18:14:48.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.779+0000 7fa78effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.784 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.780+0000 7fa78effd640 10 client.?.objecter ms_handle_connect 0x7fa7980b4e70 2024-03-25T18:14:48.784 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.780+0000 7fa78effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.784 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.780+0000 7fa78effd640 10 client.?.objecter ms_handle_connect 0x7fa7980b44b0 2024-03-25T18:14:48.784 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.780+0000 7fa78effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:48.784 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49565219 0 0) 0x7fa798127fd0 con 0x7fa7980b1b00 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): my global_id is 4224 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa77c004eb0 con 0x7fa7980b1b00 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957562074 0 0) 0x7fa79813c3d0 con 0x7fa7980b4e70 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 10 monclient(hunting): my global_id is 4229 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.782+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa798127fd0 con 0x7fa7980b4e70 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.783+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 641893738 0 0) 0x7fa7981dbc80 con 0x7fa7980b44b0 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.783+0000 7fa78effd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.783+0000 7fa78effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.783+0000 7fa78effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:48.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.783+0000 7fa78effd640 10 monclient(hunting): my global_id is 4216 2024-03-25T18:14:48.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.783+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa79813c3d0 con 0x7fa7980b44b0 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.787+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2882991728 0 0) 0x7fa77c004eb0 con 0x7fa7980b1b00 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.787+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa7981dbc80 con 0x7fa7980b1b00 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.787+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa7940019d0 con 0x7fa7980b1b00 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.787+0000 7fa78effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.787+0000 7fa78effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.787+0000 7fa78effd640 10 monclient(hunting): dump: 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:48.788 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:48.789 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:48.789 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:48.789 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:48.789 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.787+0000 7fa78effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:48.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.790+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2990116527 0 0) 0x7fa79813c3d0 con 0x7fa7980b44b0 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.790+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa77c004eb0 con 0x7fa7980b44b0 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.790+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa788001960 con 0x7fa7980b44b0 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.790+0000 7fa78effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.790+0000 7fa78effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.790+0000 7fa78effd640 10 monclient(hunting): dump: 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:48.793 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1093775504 0 0) 0x7fa7981dbc80 con 0x7fa7980b1b00 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 >> v1:172.21.15.7:6790/0 conn(0x7fa7980b4e70 legacy=0x7fa7981db570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 >> v1:172.21.15.183:6789/0 conn(0x7fa7980b44b0 legacy=0x7fa798144e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 1 monclient: found mon.a 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa79813c6b0 con 0x7fa7980b1b00 2024-03-25T18:14:48.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 10 monclient: _finish_auth 0 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa78effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 5 monclient: authenticate success, global_id 4224 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 10 monclient: _renew_subs 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa79813c990 con 0x7fa7980b1b00 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 10 client.4224.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 10 monclient: _renew_subs 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa798140560 con 0x7fa7980b1b00 2024-03-25T18:14:48.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.791+0000 7fa7a037b640 1 librados: init done 2024-03-25T18:14:48.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.792+0000 7fa7a037b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.792+0000 7fa7a037b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:48.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.792+0000 7fa7a037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:48.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:48.792+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa79813ceb0 con 0x7fa7980b1b00 2024-03-25T18:14:49.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.157+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa7940035f0 con 0x7fa7980b1b00 2024-03-25T18:14:49.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.157+0000 7fa78effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:49.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.157+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa7940039c0 con 0x7fa7980b1b00 2024-03-25T18:14:49.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.157+0000 7fa78effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:49.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.157+0000 7fa78effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:14:49.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.157+0000 7fa78effd640 10 monclient: dump: 2024-03-25T18:14:49.158 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:49.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.157+0000 7fa79c914640 4 set_mon_vals no callback set 2024-03-25T18:14:49.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.158+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fa794003be0 con 0x7fa7980b1b00 2024-03-25T18:14:49.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.158+0000 7fa79d916640 10 monclient: get_auth_request con 0x7fa77c06dfb0 auth_method 0 2024-03-25T18:14:49.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.159+0000 7fa78effd640 10 client.4224.objecter ms_handle_connect 0x7fa77c06dfb0 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fa794054850 con 0x7fa7980b1b00 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 10 client.4224.objecter ms_dispatch 0x7fa798000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 3 client.4224.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 3 client.4224.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 20 client.4224.objecter dump_active .. 0 homeless 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa7940501d0 con 0x7fa7980b1b00 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.172+0000 7fa78effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:49.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.401+0000 7fa7a037b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:49.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.401+0000 7fa7a037b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:49.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.401+0000 7fa7a037b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:14:49.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.401+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa79804f730 con 0x7fa7980b1b00 2024-03-25T18:14:49.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.503+0000 7fa78effd640 1 -- 172.21.15.7:0/3362144258 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fa794052290 con 0x7fa7980b1b00 2024-03-25T18:14:49.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.503+0000 7fa78effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:49.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.503+0000 7fa78effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:49.504 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:14:49.504 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 10 librados: watch_flush enter 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 10 librados: watch_flush exit 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 20 client.4224.objecter shutdown clearing up homeless session... 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 10 client.4224.objecter successfully canceled tick 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 >> v1:172.21.15.7:6800/1717564767 conn(0x7fa77c06dfb0 legacy=0x7fa77c070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 10 monclient: shutdown 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 >> v1:172.21.15.7:6789/0 conn(0x7fa7980b1b00 legacy=0x7fa7981dae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 shutdown_connections 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.506+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 >> 172.21.15.7:0/3362144258 conn(0x7fa7980617a0 msgr2=0x7fa7980b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:49.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.507+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 shutdown_connections 2024-03-25T18:14:49.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.507+0000 7fa7a037b640 1 -- 172.21.15.7:0/3362144258 wait complete. 2024-03-25T18:14:49.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:49.507+0000 7fa7a037b640 1 librados: shutdown 2024-03-25T18:14:49.521 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:14:51.161 INFO:tasks.ceph.osd.6.smithi183.stderr:2024-03-25T18:14:51.157+0000 5dc3480 -1 osd.6 0 log_to_monitors true 2024-03-25T18:14:51.174 INFO:tasks.ceph.osd.7.smithi183.stderr:2024-03-25T18:14:51.171+0000 5dc3480 -1 osd.7 0 log_to_monitors true 2024-03-25T18:14:51.523 INFO:tasks.ceph.osd.4.smithi183.stderr:2024-03-25T18:14:51.519+0000 5dc3480 -1 osd.4 0 log_to_monitors true 2024-03-25T18:14:51.727 INFO:tasks.ceph.osd.5.smithi183.stderr:2024-03-25T18:14:51.722+0000 5dc3480 -1 osd.5 0 log_to_monitors true 2024-03-25T18:14:52.492 INFO:tasks.ceph.osd.0.smithi007.stderr:2024-03-25T18:14:52.488+0000 5dc3480 -1 osd.0 0 log_to_monitors true 2024-03-25T18:14:53.103 INFO:tasks.ceph.osd.2.smithi007.stderr:2024-03-25T18:14:53.099+0000 5dc3480 -1 osd.2 0 log_to_monitors true 2024-03-25T18:14:53.715 INFO:tasks.ceph.osd.3.smithi007.stderr:2024-03-25T18:14:53.711+0000 5dc3480 -1 osd.3 0 log_to_monitors true 2024-03-25T18:14:54.244 INFO:tasks.ceph.osd.1.smithi007.stderr:2024-03-25T18:14:54.240+0000 5dc3480 -1 osd.1 0 log_to_monitors true 2024-03-25T18:14:56.328 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:14:56.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.443+0000 7f8e50dc9640 10 monclient: get_monmap_and_config 2024-03-25T18:14:56.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.443+0000 7f8e50dc9640 10 monclient: build_initial_monmap 2024-03-25T18:14:56.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.443+0000 7f8e50dc9640 10 monclient: monmap: 2024-03-25T18:14:56.444 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:56.444 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:14:56.444568+0000 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:14:56.444568+0000 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:14:56.445 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:56.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.444+0000 7f8e50dc9640 1 Processor -- start 2024-03-25T18:14:56.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.444+0000 7f8e50dc9640 1 -- start start 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.444+0000 7f8e50dc9640 10 monclient: init 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4c0b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4c0b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4c13c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0ab0e0 con 0x7f8e4c13c480 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0ab3c0 con 0x7f8e4c0b4ff0 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0a9f30 con 0x7f8e4c0b4670 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:56.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.445+0000 7f8e50dc9640 10 monclient(hunting): authenticate will time out at 7338.550249s 2024-03-25T18:14:56.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.456+0000 7f8e4a59c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f8e4c0b4ff0 0x7f8e4c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:34938/0 (socket says 172.21.15.7:34938) 2024-03-25T18:14:56.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.456+0000 7f8e4a59c640 1 -- 172.21.15.7:0/4058249753 learned_addr learned my addr 172.21.15.7:0/4058249753 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:56.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.468+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 51930731 0 0) 0x7f8e4c0a9f30 con 0x7f8e4c0b4670 2024-03-25T18:14:56.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.468+0000 7f8e49d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:14:56.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.468+0000 7f8e49d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:56.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.468+0000 7f8e49d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:56.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.468+0000 7f8e49d9b640 10 monclient(hunting): my global_id is 4256 2024-03-25T18:14:56.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.468+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e38003520 con 0x7f8e4c0b4670 2024-03-25T18:14:56.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 268804430 0 0) 0x7f8e38003520 con 0x7f8e4c0b4670 2024-03-25T18:14:56.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 >> v1:172.21.15.183:6789/0 conn(0x7f8e4c0b4ff0 legacy=0x7f8e4c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 >> v1:172.21.15.7:6789/0 conn(0x7f8e4c13c480 legacy=0x7f8e4c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 1 monclient: found mon.noname-b 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0a9b70 con 0x7f8e4c0b4670 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 10 monclient: _finish_auth 0 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8e34001690 con 0x7f8e4c0b4670 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e49d9b640 10 monclient: dump: 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:56.482 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e50dc9640 5 monclient: authenticate success, global_id 4256 2024-03-25T18:14:56.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.480+0000 7f8e50dc9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:14:56.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e340017f0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e49d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:56.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e49d9b640 1 -- 172.21.15.7:0/4058249753 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8e34001c40 con 0x7f8e4c0b4670 2024-03-25T18:14:56.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e49d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:56.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e49d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:56.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e49d9b640 10 monclient: dump: 2024-03-25T18:14:56.559 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e50dc9640 10 monclient: get_monmap_and_config success 2024-03-25T18:14:56.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.558+0000 7f8e50dc9640 4 set_mon_vals no callback set 2024-03-25T18:14:56.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.559+0000 7f8e50dc9640 10 monclient: shutdown 2024-03-25T18:14:56.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.559+0000 7f8e50dc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:56.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.559+0000 7f8e50dc9640 1 -- 172.21.15.7:0/4058249753 >> v1:172.21.15.7:6790/0 conn(0x7f8e4c0b4670 legacy=0x7f8e4c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.559+0000 7f8e50dc9640 1 -- 172.21.15.7:0/4058249753 shutdown_connections 2024-03-25T18:14:56.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.559+0000 7f8e50dc9640 1 -- 172.21.15.7:0/4058249753 >> 172.21.15.7:0/4058249753 conn(0x7f8e4c0617a0 msgr2=0x7f8e4c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:56.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.560+0000 7f8e50dc9640 1 -- 172.21.15.7:0/4058249753 shutdown_connections 2024-03-25T18:14:56.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.560+0000 7f8e50dc9640 1 -- 172.21.15.7:0/4058249753 wait complete. 2024-03-25T18:14:56.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 10 monclient: build_initial_monmap 2024-03-25T18:14:56.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 10 monclient: monmap: 2024-03-25T18:14:56.562 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:14:56.562 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:14:56.562 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:14:56.562 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 1 librados: starting msgr at 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 1 librados: starting objecter 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 1 Processor -- start 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 1 -- start start 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 1 librados: setting wanted keys 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 1 librados: calling monclient init 2024-03-25T18:14:56.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.561+0000 7f8e50dc9640 10 monclient: init 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 10 monclient: _reopen_session rank -1 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 10 monclient(hunting): picked mon.noname-c con 0x7f8e4c0b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 10 monclient(hunting): picked mon.noname-a con 0x7f8e4c0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 10 monclient(hunting): picked mon.noname-b con 0x7f8e4c13c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c13b120 con 0x7f8e4c0b4ff0 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0b1990 con 0x7f8e4c13c480 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8e4c0b0130 con 0x7f8e4c0b4670 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 10 monclient(hunting): _renew_subs 2024-03-25T18:14:56.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.562+0000 7f8e50dc9640 10 monclient(hunting): authenticate will time out at 7338.667067s 2024-03-25T18:14:56.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.576+0000 7f8e4a59c640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f8e4c0b4670 0x7f8e4c0b0e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:51034/0 (socket says 172.21.15.7:51034) 2024-03-25T18:14:56.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.576+0000 7f8e4a59c640 1 -- 172.21.15.7:0/158471115 learned_addr learned my addr 172.21.15.7:0/158471115 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:14:56.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.579+0000 7f8e42ffd640 10 client.?.objecter ms_handle_connect 0x7f8e4c0b4670 2024-03-25T18:14:56.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.579+0000 7f8e42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:56.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.596+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010092792 0 0) 0x7f8e4c0b0130 con 0x7f8e4c0b4670 2024-03-25T18:14:56.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.596+0000 7f8e42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:56.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.596+0000 7f8e42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:56.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.596+0000 7f8e42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:56.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.596+0000 7f8e42ffd640 10 monclient(hunting): my global_id is 4259 2024-03-25T18:14:56.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.596+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c004eb0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.613+0000 7f8e42ffd640 10 client.?.objecter ms_handle_connect 0x7f8e4c13c480 2024-03-25T18:14:56.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.613+0000 7f8e42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:56.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.619+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 768204740 0 0) 0x7f8e4c0b1990 con 0x7f8e4c13c480 2024-03-25T18:14:56.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.619+0000 7f8e42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:56.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.619+0000 7f8e42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:56.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.619+0000 7f8e42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:56.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.619+0000 7f8e42ffd640 10 monclient(hunting): my global_id is 4243 2024-03-25T18:14:56.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.619+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e4c0b0130 con 0x7f8e4c13c480 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.622+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1331322402 0 0) 0x7f8e2c004eb0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.622+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e4c0b1990 con 0x7f8e4c0b4670 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.622+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8e3c0018b0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.622+0000 7f8e42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.622+0000 7f8e42ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.622+0000 7f8e42ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:56.623 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.622+0000 7f8e42ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.624+0000 7f8e42ffd640 10 client.?.objecter ms_handle_connect 0x7f8e4c0b4ff0 2024-03-25T18:14:56.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.624+0000 7f8e42ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:14:56.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482179044 0 0) 0x7f8e4c13b120 con 0x7f8e4c0b4ff0 2024-03-25T18:14:56.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:14:56.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:14:56.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:14:56.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 10 monclient(hunting): my global_id is 4251 2024-03-25T18:14:56.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8e2c004eb0 con 0x7f8e4c0b4ff0 2024-03-25T18:14:56.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1235521327 0 0) 0x7f8e4c0b0130 con 0x7f8e4c13c480 2024-03-25T18:14:56.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8e4c13b120 con 0x7f8e4c13c480 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8e44001b90 con 0x7f8e4c13c480 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.635+0000 7f8e42ffd640 10 monclient(hunting): dump: 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:56.637 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:56.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3430256606 0 0) 0x7f8e4c0b1990 con 0x7f8e4c0b4670 2024-03-25T18:14:56.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 >> v1:172.21.15.183:6789/0 conn(0x7f8e4c13c480 legacy=0x7f8e4c0aeaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 >> v1:172.21.15.7:6789/0 conn(0x7f8e4c0b4ff0 legacy=0x7f8e4c0ae390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:56.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:14:56.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 1 monclient: found mon.c 2024-03-25T18:14:56.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:14:56.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4c0b0410 con 0x7f8e4c0b4670 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 10 monclient: _finish_auth 0 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e50dc9640 5 monclient: authenticate success, global_id 4259 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e50dc9640 10 monclient: _renew_subs 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e50dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.636+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e4c0b06f0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.637+0000 7f8e50dc9640 10 client.4259.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.637+0000 7f8e50dc9640 10 monclient: _renew_subs 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.637+0000 7f8e50dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.637+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e4c1e35a0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.637+0000 7f8e50dc9640 1 librados: init done 2024-03-25T18:14:56.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.638+0000 7f8e50dc9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.638+0000 7f8e50dc9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:56.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.638+0000 7f8e50dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:56.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.638+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e4c0688c0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.992+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8e3c0047e0 con 0x7f8e4c0b4670 2024-03-25T18:14:56.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.992+0000 7f8e42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:14:56.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.992+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8e3c004c30 con 0x7f8e4c0b4670 2024-03-25T18:14:56.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.992+0000 7f8e42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:14:56.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.992+0000 7f8e42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:14:56.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.992+0000 7f8e42ffd640 10 monclient: dump: 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:14:56.994 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.992+0000 7f8e4959a640 4 set_mon_vals no callback set 2024-03-25T18:14:56.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.993+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f8e3c004e50 con 0x7f8e4c0b4670 2024-03-25T18:14:56.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.994+0000 7f8e43fff640 10 monclient: get_auth_request con 0x7f8e2c06df70 auth_method 0 2024-03-25T18:14:56.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:56.994+0000 7f8e42ffd640 10 client.4259.objecter ms_handle_connect 0x7f8e2c06df70 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f8e3c0547c0 con 0x7f8e4c0b4670 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 10 client.4259.objecter ms_dispatch 0x7f8e4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 3 client.4259.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 3 client.4259.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 20 client.4259.objecter dump_active .. 0 homeless 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8e3c0502d0 con 0x7f8e4c0b4670 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:14:57.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.000+0000 7f8e42ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:14:57.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.258+0000 7f8e50dc9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:57.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.258+0000 7f8e50dc9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:57.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.258+0000 7f8e50dc9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:14:57.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.258+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8e4c06dbd0 con 0x7f8e4c0b4670 2024-03-25T18:14:57.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.307+0000 7f8e42ffd640 1 -- 172.21.15.7:0/158471115 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f8e3c052200 con 0x7f8e4c0b4670 2024-03-25T18:14:57.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.307+0000 7f8e42ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:14:57.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.307+0000 7f8e42ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:14:57.308 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:14:57.309 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 10 librados: watch_flush enter 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 10 librados: watch_flush exit 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 20 client.4259.objecter shutdown clearing up homeless session... 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 10 client.4259.objecter successfully canceled tick 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 >> v1:172.21.15.7:6800/1717564767 conn(0x7f8e2c06df70 legacy=0x7f8e2c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 10 monclient: shutdown 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 >> v1:172.21.15.7:6790/0 conn(0x7f8e4c0b4670 legacy=0x7f8e4c0b0e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 shutdown_connections 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.309+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 >> 172.21.15.7:0/158471115 conn(0x7f8e4c0617a0 msgr2=0x7f8e4c145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.310+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 shutdown_connections 2024-03-25T18:14:57.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.310+0000 7f8e50dc9640 1 -- 172.21.15.7:0/158471115 wait complete. 2024-03-25T18:14:57.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:14:57.310+0000 7f8e50dc9640 1 librados: shutdown 2024-03-25T18:14:57.322 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:04.129 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:04.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.245+0000 7fb864205640 10 monclient: get_monmap_and_config 2024-03-25T18:15:04.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.245+0000 7fb864205640 10 monclient: build_initial_monmap 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.245+0000 7fb864205640 10 monclient: monmap: 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:04.246112+0000 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:04.246112+0000 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:04.246 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.245+0000 7fb864205640 1 Processor -- start 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.246+0000 7fb864205640 1 -- start start 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.246+0000 7fb864205640 10 monclient: init 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.246+0000 7fb864205640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.246+0000 7fb864205640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.246+0000 7fb864205640 10 monclient(hunting): picked mon.noname-a con 0x7fb85c0b1a70 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.247+0000 7fb864205640 10 monclient(hunting): picked mon.noname-b con 0x7fb85c1405b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.247+0000 7fb864205640 10 monclient(hunting): picked mon.noname-c con 0x7fb85c13d690 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.247+0000 7fb864205640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c072990 con 0x7fb85c0b1a70 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.247+0000 7fb864205640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c072c70 con 0x7fb85c13d690 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.247+0000 7fb864205640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c13dc30 con 0x7fb85c1405b0 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.247+0000 7fb864205640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:04.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.247+0000 7fb864205640 10 monclient(hunting): authenticate will time out at 7346.351879s 2024-03-25T18:15:04.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.261+0000 7fb8627a2640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb85c13d690 0x7fb85c14ba10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:58754/0 (socket says 172.21.15.7:58754) 2024-03-25T18:15:04.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.261+0000 7fb8627a2640 1 -- 172.21.15.7:0/1444970842 learned_addr learned my addr 172.21.15.7:0/1444970842 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:04.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.271+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3618681867 0 0) 0x7fb85c072990 con 0x7fb85c0b1a70 2024-03-25T18:15:04.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.271+0000 7fb860f9f640 10 monclient(hunting): _negotiate 2024-03-25T18:15:04.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.271+0000 7fb860f9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:04.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.271+0000 7fb860f9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:04.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.271+0000 7fb860f9f640 10 monclient(hunting): my global_id is 4254 2024-03-25T18:15:04.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.271+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb848003520 con 0x7fb85c0b1a70 2024-03-25T18:15:04.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.274+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3413974141 0 0) 0x7fb85c072c70 con 0x7fb85c13d690 2024-03-25T18:15:04.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.274+0000 7fb860f9f640 10 monclient(hunting): _negotiate 2024-03-25T18:15:04.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.274+0000 7fb860f9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:04.275 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.274+0000 7fb860f9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:04.275 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.274+0000 7fb860f9f640 10 monclient(hunting): my global_id is 4246 2024-03-25T18:15:04.275 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.274+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c072990 con 0x7fb85c13d690 2024-03-25T18:15:04.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.277+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1569789713 0 0) 0x7fb85c13dc30 con 0x7fb85c1405b0 2024-03-25T18:15:04.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.277+0000 7fb860f9f640 10 monclient(hunting): _negotiate 2024-03-25T18:15:04.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.277+0000 7fb860f9f640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:04.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.277+0000 7fb860f9f640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:04.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.277+0000 7fb860f9f640 10 monclient(hunting): my global_id is 4262 2024-03-25T18:15:04.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.277+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c072c70 con 0x7fb85c1405b0 2024-03-25T18:15:04.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3393923529 0 0) 0x7fb848003520 con 0x7fb85c0b1a70 2024-03-25T18:15:04.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 >> v1:172.21.15.7:6790/0 conn(0x7fb85c1405b0 legacy=0x7fb85c06ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 >> v1:172.21.15.183:6789/0 conn(0x7fb85c13d690 legacy=0x7fb85c14ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 10 monclient: _finish_hunting 0 2024-03-25T18:15:04.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 1 monclient: found mon.noname-a 2024-03-25T18:15:04.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:04.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c140b20 con 0x7fb85c0b1a70 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 10 monclient: _finish_auth 0 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 10 monclient: _check_auth_tickets 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb858001690 con 0x7fb85c0b1a70 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb864205640 5 monclient: authenticate success, global_id 4254 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb864205640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb860f9f640 10 monclient: dump: 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:04.283 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:04.284 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:04.284 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:04.284 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:04.284 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.281+0000 7fb864205640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8580017f0 con 0x7fb85c0b1a70 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb860f9f640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb860f9f640 1 -- 172.21.15.7:0/1444970842 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb858001c00 con 0x7fb85c0b1a70 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb860f9f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb860f9f640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb860f9f640 10 monclient: dump: 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.295 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb864205640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:04.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.294+0000 7fb864205640 4 set_mon_vals no callback set 2024-03-25T18:15:04.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.295+0000 7fb864205640 10 monclient: shutdown 2024-03-25T18:15:04.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.295+0000 7fb864205640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:04.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.295+0000 7fb864205640 1 -- 172.21.15.7:0/1444970842 >> v1:172.21.15.7:6789/0 conn(0x7fb85c0b1a70 legacy=0x7fb85c13cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.295+0000 7fb864205640 1 -- 172.21.15.7:0/1444970842 shutdown_connections 2024-03-25T18:15:04.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.295+0000 7fb864205640 1 -- 172.21.15.7:0/1444970842 >> 172.21.15.7:0/1444970842 conn(0x7fb85c0b28f0 msgr2=0x7fb85c0b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:04.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.295+0000 7fb864205640 1 -- 172.21.15.7:0/1444970842 shutdown_connections 2024-03-25T18:15:04.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.295+0000 7fb864205640 1 -- 172.21.15.7:0/1444970842 wait complete. 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.296+0000 7fb864205640 10 monclient: build_initial_monmap 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.296+0000 7fb864205640 10 monclient: monmap: 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.296+0000 7fb864205640 1 librados: starting msgr at 2024-03-25T18:15:04.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.296+0000 7fb864205640 1 librados: starting objecter 2024-03-25T18:15:04.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.296+0000 7fb864205640 1 Processor -- start 2024-03-25T18:15:04.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 1 -- start start 2024-03-25T18:15:04.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 1 librados: setting wanted keys 2024-03-25T18:15:04.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 1 librados: calling monclient init 2024-03-25T18:15:04.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient: init 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient(hunting): picked mon.noname-a con 0x7fb85c0b1a70 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient(hunting): picked mon.noname-c con 0x7fb85c13d690 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient(hunting): picked mon.noname-b con 0x7fb85c1405b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c120190 con 0x7fb85c0b1a70 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c120400 con 0x7fb85c1405b0 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb85c11f350 con 0x7fb85c13d690 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:04.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.297+0000 7fb864205640 10 monclient(hunting): authenticate will time out at 7346.402526s 2024-03-25T18:15:04.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.312+0000 7fb8627a2640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb85c1405b0 0x7fb85c1d7880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:58788/0 (socket says 172.21.15.7:58788) 2024-03-25T18:15:04.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.313+0000 7fb8627a2640 1 -- 172.21.15.7:0/735730469 learned_addr learned my addr 172.21.15.7:0/735730469 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:04.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.316+0000 7fb8527fc640 10 client.?.objecter ms_handle_connect 0x7fb85c0b1a70 2024-03-25T18:15:04.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.316+0000 7fb8527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:04.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.318+0000 7fb8527fc640 10 client.?.objecter ms_handle_connect 0x7fb85c13d690 2024-03-25T18:15:04.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.318+0000 7fb8527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:04.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.324+0000 7fb8527fc640 10 client.?.objecter ms_handle_connect 0x7fb85c1405b0 2024-03-25T18:15:04.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.324+0000 7fb8527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 424139055 0 0) 0x7fb85c120190 con 0x7fb85c0b1a70 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): my global_id is 4257 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb840004eb0 con 0x7fb85c0b1a70 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110966546 0 0) 0x7fb85c120400 con 0x7fb85c1405b0 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:04.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:04.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:04.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 10 monclient(hunting): my global_id is 4249 2024-03-25T18:15:04.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.328+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c120190 con 0x7fb85c1405b0 2024-03-25T18:15:04.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.333+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1831159541 0 0) 0x7fb85c11f350 con 0x7fb85c13d690 2024-03-25T18:15:04.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.333+0000 7fb8527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:04.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.333+0000 7fb8527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:04.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.333+0000 7fb8527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:04.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.333+0000 7fb8527fc640 10 monclient(hunting): my global_id is 4265 2024-03-25T18:15:04.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.333+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb85c120400 con 0x7fb85c13d690 2024-03-25T18:15:04.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.339+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 879908706 0 0) 0x7fb840004eb0 con 0x7fb85c0b1a70 2024-03-25T18:15:04.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.339+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb85c11f350 con 0x7fb85c0b1a70 2024-03-25T18:15:04.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.339+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb858001b00 con 0x7fb85c0b1a70 2024-03-25T18:15:04.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.339+0000 7fb8527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:04.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.339+0000 7fb8527fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.339+0000 7fb8527fc640 10 monclient(hunting): dump: 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.339+0000 7fb8527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.340+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2463590141 0 0) 0x7fb85c120190 con 0x7fb85c1405b0 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.340+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb840004eb0 con 0x7fb85c1405b0 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.340+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb854001c70 con 0x7fb85c1405b0 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.340+0000 7fb8527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.340+0000 7fb8527fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.340+0000 7fb8527fc640 10 monclient(hunting): dump: 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.342 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.343 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:04.343 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:04.343 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:04.343 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:04.343 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:04.343 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.354+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3406555456 0 0) 0x7fb85c120400 con 0x7fb85c13d690 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.354+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb85c120190 con 0x7fb85c13d690 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.354+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb84c001790 con 0x7fb85c13d690 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.354+0000 7fb8527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.354+0000 7fb8527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.354+0000 7fb8527fc640 10 monclient(hunting): dump: 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:04.355 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:04.356 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.360 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4209469707 0 0) 0x7fb85c11f350 con 0x7fb85c0b1a70 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 >> v1:172.21.15.7:6790/0 conn(0x7fb85c13d690 legacy=0x7fb85c13aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 >> v1:172.21.15.183:6789/0 conn(0x7fb85c1405b0 legacy=0x7fb85c1d7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 1 monclient: found mon.a 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb85c11ed00 con 0x7fb85c0b1a70 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb8527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb864205640 5 monclient: authenticate success, global_id 4257 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb864205640 10 monclient: _renew_subs 2024-03-25T18:15:04.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb864205640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.359+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb85c11ef50 con 0x7fb85c0b1a70 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.360+0000 7fb864205640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.360+0000 7fb864205640 10 monclient: _renew_subs 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.360+0000 7fb864205640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.360+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb85c13b790 con 0x7fb85c0b1a70 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.360+0000 7fb864205640 1 librados: init done 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.361+0000 7fb864205640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:04.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.361+0000 7fb864205640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:04.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.361+0000 7fb864205640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:04.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.361+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb85c13b9e0 con 0x7fb85c0b1a70 2024-03-25T18:15:04.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.663+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb8580038a0 con 0x7fb85c0b1a70 2024-03-25T18:15:04.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.663+0000 7fb8527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:04.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.663+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb858003bb0 con 0x7fb85c0b1a70 2024-03-25T18:15:04.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.663+0000 7fb8527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:04.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.663+0000 7fb8527fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:04.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.663+0000 7fb8527fc640 10 monclient: dump: 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.663+0000 7fb853fff640 4 set_mon_vals no callback set 2024-03-25T18:15:04.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.664+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fb858019b80 con 0x7fb85c0b1a70 2024-03-25T18:15:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.665+0000 7fb8617a0640 10 monclient: get_auth_request con 0x7fb84006e0f0 auth_method 0 2024-03-25T18:15:04.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.665+0000 7fb8527fc640 10 client.4257.objecter ms_handle_connect 0x7fb84006e0f0 2024-03-25T18:15:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fb858053650 con 0x7fb85c0b1a70 2024-03-25T18:15:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 10 client.4257.objecter ms_dispatch 0x7fb85c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:04.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-25T18:15:04.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb85804f160 con 0x7fb85c0b1a70 2024-03-25T18:15:04.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:04.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.672+0000 7fb8527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:04.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.904+0000 7fb864205640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.904+0000 7fb864205640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.904+0000 7fb864205640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:04.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.904+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb85c072f10 con 0x7fb85c0b1a70 2024-03-25T18:15:04.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.968+0000 7fb8527fc640 1 -- 172.21.15.7:0/735730469 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fb858051090 con 0x7fb85c0b1a70 2024-03-25T18:15:04.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.968+0000 7fb8527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:04.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.968+0000 7fb8527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:04.969 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:04.969 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:04.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 10 librados: watch_flush enter 2024-03-25T18:15:04.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 10 librados: watch_flush exit 2024-03-25T18:15:04.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-25T18:15:04.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 10 client.4257.objecter successfully canceled tick 2024-03-25T18:15:04.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 >> v1:172.21.15.7:6800/1717564767 conn(0x7fb84006e0f0 legacy=0x7fb8400705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 10 monclient: shutdown 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 >> v1:172.21.15.7:6789/0 conn(0x7fb85c0b1a70 legacy=0x7fb85c11e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 shutdown_connections 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 >> 172.21.15.7:0/735730469 conn(0x7fb85c0b28f0 msgr2=0x7fb85c13c130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 shutdown_connections 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 1 -- 172.21.15.7:0/735730469 wait complete. 2024-03-25T18:15:04.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:04.971+0000 7fb864205640 1 librados: shutdown 2024-03-25T18:15:04.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:08.070 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:14.707 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.132 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:14.769 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.162 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:14.789 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.238 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:14.864 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.248 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:14.885 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.308 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:14.945 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.390 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:15.027 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.459 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:15.086 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:08.543 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:15.180 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:11.791 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:11.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.909+0000 7f8885d3d640 10 monclient: get_monmap_and_config 2024-03-25T18:15:11.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.909+0000 7f8885d3d640 10 monclient: build_initial_monmap 2024-03-25T18:15:11.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.909+0000 7f8885d3d640 10 monclient: monmap: 2024-03-25T18:15:11.910 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:11.910 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:11.910 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:11.910610+0000 2024-03-25T18:15:11.910 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:11.910610+0000 2024-03-25T18:15:11.911 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:11.911 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:11.911 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:11.911 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:11.911 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:11.911 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.910+0000 7f8885d3d640 1 Processor -- start 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.910+0000 7f8885d3d640 1 -- start start 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.910+0000 7f8885d3d640 10 monclient: init 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8880142e80 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f88800af5d0 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f88800b0220 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88800afb40 con 0x7f8880142e80 2024-03-25T18:15:11.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88800b1210 con 0x7f88800af5d0 2024-03-25T18:15:11.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8880146220 con 0x7f88800b0220 2024-03-25T18:15:11.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:11.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.911+0000 7f8885d3d640 10 monclient(hunting): authenticate will time out at 7354.016148s 2024-03-25T18:15:11.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.930+0000 7f887ffff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f88800b0220 0x7f888014dbd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:34790/0 (socket says 172.21.15.7:34790) 2024-03-25T18:15:11.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.930+0000 7f887ffff640 1 -- 172.21.15.7:0/2846941681 learned_addr learned my addr 172.21.15.7:0/2846941681 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:11.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.951+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4157404883 0 0) 0x7f88800b1210 con 0x7f88800af5d0 2024-03-25T18:15:11.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.951+0000 7f887e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.951+0000 7f887e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.951+0000 7f887e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.951+0000 7f887e7fc640 10 monclient(hunting): my global_id is 4252 2024-03-25T18:15:11.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.951+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8868003520 con 0x7f88800af5d0 2024-03-25T18:15:11.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.952+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2177032017 0 0) 0x7f8880146220 con 0x7f88800b0220 2024-03-25T18:15:11.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.952+0000 7f887e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.952+0000 7f887e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.952+0000 7f887e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.952+0000 7f887e7fc640 10 monclient(hunting): my global_id is 4268 2024-03-25T18:15:11.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.952+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88800b1210 con 0x7f88800b0220 2024-03-25T18:15:11.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.961+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1661265048 0 0) 0x7f88800afb40 con 0x7f8880142e80 2024-03-25T18:15:11.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.961+0000 7f887e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:11.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.961+0000 7f887e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:11.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.961+0000 7f887e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:11.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.961+0000 7f887e7fc640 10 monclient(hunting): my global_id is 4260 2024-03-25T18:15:11.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.961+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8880146220 con 0x7f8880142e80 2024-03-25T18:15:11.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2550979597 0 0) 0x7f8868003520 con 0x7f88800af5d0 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 >> v1:172.21.15.7:6790/0 conn(0x7f88800b0220 legacy=0x7f888014dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 >> v1:172.21.15.7:6789/0 conn(0x7f8880142e80 legacy=0x7f888006aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88801463b0 con 0x7f88800af5d0 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:11.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8874001690 con 0x7f88800af5d0 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.966+0000 7f887e7fc640 10 monclient: dump: 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:11.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.967+0000 7f8885d3d640 5 monclient: authenticate success, global_id 4252 2024-03-25T18:15:11.970 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.967+0000 7f8885d3d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:11.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88740017f0 con 0x7f88800af5d0 2024-03-25T18:15:11.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f887e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:11.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f887e7fc640 1 -- 172.21.15.7:0/2846941681 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8874001c00 con 0x7f88800af5d0 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f8885d3d640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f8885d3d640 4 set_mon_vals no callback set 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f887e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f887e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f887e7fc640 10 monclient: dump: 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:11.979 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.978+0000 7f8885d3d640 10 monclient: shutdown 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 1 -- 172.21.15.7:0/2846941681 >> v1:172.21.15.183:6789/0 conn(0x7f88800af5d0 legacy=0x7f8880145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 1 -- 172.21.15.7:0/2846941681 shutdown_connections 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 1 -- 172.21.15.7:0/2846941681 >> 172.21.15.7:0/2846941681 conn(0x7f88800b4bd0 msgr2=0x7f88800b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 1 -- 172.21.15.7:0/2846941681 shutdown_connections 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 1 -- 172.21.15.7:0/2846941681 wait complete. 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 10 monclient: build_initial_monmap 2024-03-25T18:15:11.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.979+0000 7f8885d3d640 10 monclient: monmap: 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.980+0000 7f8885d3d640 1 librados: starting msgr at 2024-03-25T18:15:11.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.980+0000 7f8885d3d640 1 librados: starting objecter 2024-03-25T18:15:11.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.980+0000 7f8885d3d640 1 Processor -- start 2024-03-25T18:15:11.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.980+0000 7f8885d3d640 1 -- start start 2024-03-25T18:15:11.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.980+0000 7f8885d3d640 1 librados: setting wanted keys 2024-03-25T18:15:11.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.980+0000 7f8885d3d640 1 librados: calling monclient init 2024-03-25T18:15:11.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.980+0000 7f8885d3d640 10 monclient: init 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 10 monclient(hunting): picked mon.noname-b con 0x7f88800af5d0 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 10 monclient(hunting): picked mon.noname-c con 0x7f88800b0220 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 10 monclient(hunting): picked mon.noname-a con 0x7f8880142e80 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888013e050 con 0x7f8880142e80 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f888014e630 con 0x7f88800af5d0 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88801daf40 con 0x7f88800b0220 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:11.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.981+0000 7f8885d3d640 10 monclient(hunting): authenticate will time out at 7354.086102s 2024-03-25T18:15:11.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.997+0000 7f887effd640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f88800b0220 0x7f88801d78a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:34852/0 (socket says 172.21.15.7:34852) 2024-03-25T18:15:11.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.997+0000 7f887effd640 1 -- 172.21.15.7:0/1259330127 learned_addr learned my addr 172.21.15.7:0/1259330127 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.998+0000 7f8867fff640 10 client.?.objecter ms_handle_connect 0x7f88800b0220 2024-03-25T18:15:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:11.999+0000 7f8867fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:12.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.007+0000 7f8867fff640 10 client.?.objecter ms_handle_connect 0x7f8880142e80 2024-03-25T18:15:12.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.007+0000 7f8867fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:12.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.010+0000 7f8867fff640 10 client.?.objecter ms_handle_connect 0x7f88800af5d0 2024-03-25T18:15:12.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.010+0000 7f8867fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:12.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.015+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2127937979 0 0) 0x7f88801daf40 con 0x7f88800b0220 2024-03-25T18:15:12.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.015+0000 7f8867fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.015+0000 7f8867fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.015+0000 7f8867fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.015+0000 7f8867fff640 10 monclient(hunting): my global_id is 4271 2024-03-25T18:15:12.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.015+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f886c004eb0 con 0x7f88800b0220 2024-03-25T18:15:12.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.020+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 37047738 0 0) 0x7f888014e630 con 0x7f88800af5d0 2024-03-25T18:15:12.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.020+0000 7f8867fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.020+0000 7f8867fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.020+0000 7f8867fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.020+0000 7f8867fff640 10 monclient(hunting): my global_id is 4255 2024-03-25T18:15:12.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.020+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88801daf40 con 0x7f88800af5d0 2024-03-25T18:15:12.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.036+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2919611807 0 0) 0x7f888013e050 con 0x7f8880142e80 2024-03-25T18:15:12.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.036+0000 7f8867fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:12.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.036+0000 7f8867fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:12.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.036+0000 7f8867fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:12.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.036+0000 7f8867fff640 10 monclient(hunting): my global_id is 4263 2024-03-25T18:15:12.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.036+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f888014e630 con 0x7f8880142e80 2024-03-25T18:15:12.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.039+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4256719967 0 0) 0x7f886c004eb0 con 0x7f88800b0220 2024-03-25T18:15:12.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.039+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f888013e050 con 0x7f88800b0220 2024-03-25T18:15:12.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.039+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8874001bc0 con 0x7f88800b0220 2024-03-25T18:15:12.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.039+0000 7f8867fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:12.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.039+0000 7f8867fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:12.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.039+0000 7f8867fff640 10 monclient(hunting): dump: 2024-03-25T18:15:12.040 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:12.041 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.039+0000 7f8867fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.041+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1953195041 0 0) 0x7f88801daf40 con 0x7f88800af5d0 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.041+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f886c004eb0 con 0x7f88800af5d0 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.041+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f88700017d0 con 0x7f88800af5d0 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.041+0000 7f8867fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.041+0000 7f8867fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.041+0000 7f8867fff640 10 monclient(hunting): dump: 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:12.042 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:12.043 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:12.043 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:12.043 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:12.043 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:12.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3889280894 0 0) 0x7f886c004eb0 con 0x7f88800af5d0 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 >> v1:172.21.15.7:6790/0 conn(0x7f88800b0220 legacy=0x7f88801d78a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 >> v1:172.21.15.7:6789/0 conn(0x7f8880142e80 legacy=0x7f88801d7fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 1 monclient: found mon.b 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88801dbb70 con 0x7f88800af5d0 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8867fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 5 monclient: authenticate success, global_id 4255 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 10 monclient: _renew_subs 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88801d86c0 con 0x7f88800af5d0 2024-03-25T18:15:12.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 10 client.4255.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:12.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 10 monclient: _renew_subs 2024-03-25T18:15:12.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:12.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f888013e8f0 con 0x7f88800af5d0 2024-03-25T18:15:12.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.055+0000 7f8885d3d640 1 librados: init done 2024-03-25T18:15:12.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.057+0000 7f8885d3d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:12.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.057+0000 7f8885d3d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:12.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.057+0000 7f8885d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:12.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.057+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88800ae710 con 0x7f88800af5d0 2024-03-25T18:15:12.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.328+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8870001ce0 con 0x7f88800af5d0 2024-03-25T18:15:12.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.328+0000 7f8867fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:12.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.328+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8870003790 con 0x7f88800af5d0 2024-03-25T18:15:12.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.328+0000 7f8867fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:12.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.328+0000 7f8867fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:12.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.328+0000 7f8867fff640 10 monclient: dump: 2024-03-25T18:15:12.329 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:12.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.328+0000 7f887dffb640 4 set_mon_vals no callback set 2024-03-25T18:15:12.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.329+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f88700039b0 con 0x7f88800af5d0 2024-03-25T18:15:12.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.329+0000 7f887effd640 10 monclient: get_auth_request con 0x7f886c06df70 auth_method 0 2024-03-25T18:15:12.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.329+0000 7f8867fff640 10 client.4255.objecter ms_handle_connect 0x7f886c06df70 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f8870053500 con 0x7f88800af5d0 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 10 client.4255.objecter ms_dispatch 0x7f8880000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 3 client.4255.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 3 client.4255.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 20 client.4255.objecter dump_active .. 0 homeless 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f887004f010 con 0x7f88800af5d0 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:12.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.334+0000 7f8867fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:12.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.569+0000 7f8885d3d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:12.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.569+0000 7f8885d3d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:12.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.569+0000 7f8885d3d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:12.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.569+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88800af030 con 0x7f88800af5d0 2024-03-25T18:15:12.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.625+0000 7f8867fff640 1 -- 172.21.15.7:0/1259330127 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f8870050f40 con 0x7f88800af5d0 2024-03-25T18:15:12.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.625+0000 7f8867fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:12.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.625+0000 7f8867fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:12.626 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:12.626 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:12.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 10 librados: watch_flush enter 2024-03-25T18:15:12.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 10 librados: watch_flush exit 2024-03-25T18:15:12.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 20 client.4255.objecter shutdown clearing up homeless session... 2024-03-25T18:15:12.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 10 client.4255.objecter successfully canceled tick 2024-03-25T18:15:12.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 >> v1:172.21.15.7:6800/1717564767 conn(0x7f886c06df70 legacy=0x7f886c070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 10 monclient: shutdown 2024-03-25T18:15:12.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:12.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 >> v1:172.21.15.183:6789/0 conn(0x7f88800af5d0 legacy=0x7f88801db460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:12.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 shutdown_connections 2024-03-25T18:15:12.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 >> 172.21.15.7:0/1259330127 conn(0x7f88800b4bd0 msgr2=0x7f88800b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:12.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 shutdown_connections 2024-03-25T18:15:12.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 1 -- 172.21.15.7:0/1259330127 wait complete. 2024-03-25T18:15:12.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:12.628+0000 7f8885d3d640 1 librados: shutdown 2024-03-25T18:15:12.640 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:19.447 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.560+0000 7fb6cbc40640 10 monclient: get_monmap_and_config 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.560+0000 7fb6cbc40640 10 monclient: build_initial_monmap 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.560+0000 7fb6cbc40640 10 monclient: monmap: 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:19.561445+0000 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:19.561445+0000 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:19.562 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.561+0000 7fb6cbc40640 1 Processor -- start 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.561+0000 7fb6cbc40640 1 -- start start 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.561+0000 7fb6cbc40640 10 monclient: init 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c40b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:19.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c40b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:19.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c40b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:19.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c40a96f0 con 0x7fb6c40b44b0 2024-03-25T18:15:19.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c40ab0e0 con 0x7fb6c40b4e70 2024-03-25T18:15:19.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c4145f70 con 0x7fb6c40b1b00 2024-03-25T18:15:19.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:19.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.562+0000 7fb6cbc40640 10 monclient(hunting): authenticate will time out at 7361.667063s 2024-03-25T18:15:19.574 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.573+0000 7fb6c99dc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb6c40b4e70 0x7fb6c413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:59566/0 (socket says 172.21.15.7:59566) 2024-03-25T18:15:19.574 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.573+0000 7fb6c99dc640 1 -- 172.21.15.7:0/2541915604 learned_addr learned my addr 172.21.15.7:0/2541915604 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:19.654 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.654+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168397335 0 0) 0x7fb6c4145f70 con 0x7fb6c40b1b00 2024-03-25T18:15:19.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.654+0000 7fb6c89da640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.654+0000 7fb6c89da640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.654+0000 7fb6c89da640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.654+0000 7fb6c89da640 10 monclient(hunting): my global_id is 4274 2024-03-25T18:15:19.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.654+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6b0003520 con 0x7fb6c40b1b00 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2291178014 0 0) 0x7fb6b0003520 con 0x7fb6c40b1b00 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 >> v1:172.21.15.183:6789/0 conn(0x7fb6c40b4e70 legacy=0x7fb6c413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 >> v1:172.21.15.7:6789/0 conn(0x7fb6c40b44b0 legacy=0x7fb6c4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 10 monclient: _finish_hunting 0 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 1 monclient: found mon.noname-b 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c40a9b80 con 0x7fb6c40b1b00 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 10 monclient: _finish_auth 0 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 10 monclient: _check_auth_tickets 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:19.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6bc001a10 con 0x7fb6c40b1b00 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6c89da640 10 monclient: dump: 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6cbc40640 5 monclient: authenticate success, global_id 4274 2024-03-25T18:15:19.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.668+0000 7fb6cbc40640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:19.679 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.678+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6bc001b70 con 0x7fb6c40b1b00 2024-03-25T18:15:19.679 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6c89da640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:19.679 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6c89da640 1 -- 172.21.15.7:0/2541915604 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6bc001f80 con 0x7fb6c40b1b00 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 4 set_mon_vals no callback set 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6c89da640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6c89da640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6c89da640 10 monclient: dump: 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:19.680 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 10 monclient: shutdown 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 1 -- 172.21.15.7:0/2541915604 >> v1:172.21.15.7:6790/0 conn(0x7fb6c40b1b00 legacy=0x7fb6c414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 1 -- 172.21.15.7:0/2541915604 shutdown_connections 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 1 -- 172.21.15.7:0/2541915604 >> 172.21.15.7:0/2541915604 conn(0x7fb6c40617a0 msgr2=0x7fb6c4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 1 -- 172.21.15.7:0/2541915604 shutdown_connections 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.679+0000 7fb6cbc40640 1 -- 172.21.15.7:0/2541915604 wait complete. 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.680+0000 7fb6cbc40640 10 monclient: build_initial_monmap 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.680+0000 7fb6cbc40640 10 monclient: monmap: 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:19.683 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 librados: starting msgr at 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 librados: starting objecter 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 Processor -- start 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 -- start start 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 librados: setting wanted keys 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 librados: calling monclient init 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient: init 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:19.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient(hunting): picked mon.noname-c con 0x7fb6c40b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient(hunting): picked mon.noname-b con 0x7fb6c40b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient(hunting): picked mon.noname-a con 0x7fb6c40b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c41404b0 con 0x7fb6c40b4e70 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c41435d0 con 0x7fb6c40b44b0 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6c4140160 con 0x7fb6c40b1b00 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:19.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.681+0000 7fb6cbc40640 10 monclient(hunting): authenticate will time out at 7361.786550s 2024-03-25T18:15:19.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.689+0000 7fb6c99dc640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fb6c40b1b00 0x7fb6c413b780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:39412/0 (socket says 172.21.15.7:39412) 2024-03-25T18:15:19.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.689+0000 7fb6c99dc640 1 -- 172.21.15.7:0/1377275642 learned_addr learned my addr 172.21.15.7:0/1377275642 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:19.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.695+0000 7fb6ba7fc640 10 client.?.objecter ms_handle_connect 0x7fb6c40b1b00 2024-03-25T18:15:19.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.695+0000 7fb6ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:19.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.699+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 566348391 0 0) 0x7fb6c4140160 con 0x7fb6c40b1b00 2024-03-25T18:15:19.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.699+0000 7fb6ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.699+0000 7fb6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.699+0000 7fb6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.699+0000 7fb6ba7fc640 10 monclient(hunting): my global_id is 4277 2024-03-25T18:15:19.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.699+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a8004eb0 con 0x7fb6c40b1b00 2024-03-25T18:15:19.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.712+0000 7fb6ba7fc640 10 client.?.objecter ms_handle_connect 0x7fb6c40b44b0 2024-03-25T18:15:19.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.712+0000 7fb6ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:19.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.719+0000 7fb6ba7fc640 10 client.?.objecter ms_handle_connect 0x7fb6c40b4e70 2024-03-25T18:15:19.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.719+0000 7fb6ba7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:19.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.720+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042885903 0 0) 0x7fb6c41435d0 con 0x7fb6c40b44b0 2024-03-25T18:15:19.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.720+0000 7fb6ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.720+0000 7fb6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.720+0000 7fb6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.720+0000 7fb6ba7fc640 10 monclient(hunting): my global_id is 4258 2024-03-25T18:15:19.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.720+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c4140160 con 0x7fb6c40b44b0 2024-03-25T18:15:19.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.721+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3304652271 0 0) 0x7fb6a8004eb0 con 0x7fb6c40b1b00 2024-03-25T18:15:19.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.721+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6c41435d0 con 0x7fb6c40b1b00 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.721+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6c0001810 con 0x7fb6c40b1b00 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.721+0000 7fb6ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.721+0000 7fb6ba7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.721+0000 7fb6ba7fc640 10 monclient(hunting): dump: 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:19.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.722+0000 7fb6ba7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:19.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.727+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 707813041 0 0) 0x7fb6c41404b0 con 0x7fb6c40b4e70 2024-03-25T18:15:19.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.727+0000 7fb6ba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:19.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.727+0000 7fb6ba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:19.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.727+0000 7fb6ba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:19.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.727+0000 7fb6ba7fc640 10 monclient(hunting): my global_id is 4269 2024-03-25T18:15:19.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.727+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6a8004eb0 con 0x7fb6c40b4e70 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.736+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2402838053 0 0) 0x7fb6a8004eb0 con 0x7fb6c40b4e70 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.736+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6c41404b0 con 0x7fb6c40b4e70 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.736+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6bc001e80 con 0x7fb6c40b4e70 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.736+0000 7fb6ba7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.736+0000 7fb6ba7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.736+0000 7fb6ba7fc640 10 monclient(hunting): dump: 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:19.737 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:19.738 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2061992092 0 0) 0x7fb6c41435d0 con 0x7fb6c40b1b00 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 >> v1:172.21.15.183:6789/0 conn(0x7fb6c40b44b0 legacy=0x7fb6c41df5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 >> v1:172.21.15.7:6789/0 conn(0x7fb6c40b4e70 legacy=0x7fb6c41e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 1 monclient: found mon.c 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6c41408d0 con 0x7fb6c40b1b00 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6ba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 5 monclient: authenticate success, global_id 4277 2024-03-25T18:15:19.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 10 monclient: _renew_subs 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6c4142b20 con 0x7fb6c40b1b00 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 10 monclient: _renew_subs 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6c413b060 con 0x7fb6c40b1b00 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.743+0000 7fb6cbc40640 1 librados: init done 2024-03-25T18:15:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.744+0000 7fb6cbc40640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:19.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.744+0000 7fb6cbc40640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:19.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.744+0000 7fb6cbc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:19.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.744+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6c4140a80 con 0x7fb6c40b1b00 2024-03-25T18:15:19.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.755+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6c0001d00 con 0x7fb6c40b1b00 2024-03-25T18:15:19.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.755+0000 7fb6ba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:19.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:19.755+0000 7fb6bbfff640 4 set_mon_vals no callback set 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.062+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6c00047e0 con 0x7fb6c40b1b00 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.062+0000 7fb6ba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.062+0000 7fb6ba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.062+0000 7fb6ba7fc640 10 monclient: dump: 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:20.063 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:20.064 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:20.064 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:20.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.063+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fb6c0004a00 con 0x7fb6c40b1b00 2024-03-25T18:15:20.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.063+0000 7fb6c91db640 10 monclient: get_auth_request con 0x7fb6a806df20 auth_method 0 2024-03-25T18:15:20.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.063+0000 7fb6ba7fc640 10 client.4277.objecter ms_handle_connect 0x7fb6a806df20 2024-03-25T18:15:20.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fb6c0054670 con 0x7fb6c40b1b00 2024-03-25T18:15:20.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 10 client.4277.objecter ms_dispatch 0x7fb6c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:20.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:20.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:20.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-25T18:15:20.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6c004ffb0 con 0x7fb6c40b1b00 2024-03-25T18:15:20.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:20.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.072+0000 7fb6ba7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:20.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.329+0000 7fb6cbc40640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:20.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.329+0000 7fb6cbc40640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:20.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.329+0000 7fb6cbc40640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:20.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.329+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6c4145940 con 0x7fb6c40b1b00 2024-03-25T18:15:20.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.363+0000 7fb6ba7fc640 1 -- 172.21.15.7:0/1377275642 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fb6c00520b0 con 0x7fb6c40b1b00 2024-03-25T18:15:20.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.363+0000 7fb6ba7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:20.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.363+0000 7fb6ba7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:20.364 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:20.364 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 10 librados: watch_flush enter 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 10 librados: watch_flush exit 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 10 client.4277.objecter successfully canceled tick 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 >> v1:172.21.15.7:6800/1717564767 conn(0x7fb6a806df20 legacy=0x7fb6a80703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 10 monclient: shutdown 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 >> v1:172.21.15.7:6790/0 conn(0x7fb6c40b1b00 legacy=0x7fb6c413b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 shutdown_connections 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 >> 172.21.15.7:0/1377275642 conn(0x7fb6c40617a0 msgr2=0x7fb6c40b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 shutdown_connections 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 1 -- 172.21.15.7:0/1377275642 wait complete. 2024-03-25T18:15:20.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:20.365+0000 7fb6cbc40640 1 librados: shutdown 2024-03-25T18:15:20.378 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:27.185 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:27.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.299+0000 7fb31f698640 10 monclient: get_monmap_and_config 2024-03-25T18:15:27.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.299+0000 7fb31f698640 10 monclient: build_initial_monmap 2024-03-25T18:15:27.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.300+0000 7fb31f698640 10 monclient: monmap: 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:27.301020+0000 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:27.301020+0000 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:27.301 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:27.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.300+0000 7fb31f698640 1 Processor -- start 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.300+0000 7fb31f698640 1 -- start start 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.300+0000 7fb31f698640 10 monclient: init 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 10 monclient(hunting): picked mon.noname-b con 0x7fb3180b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 10 monclient(hunting): picked mon.noname-a con 0x7fb3180b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 10 monclient(hunting): picked mon.noname-c con 0x7fb31813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180ab0e0 con 0x7fb3180b4670 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180ab3c0 con 0x7fb31813c480 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3180a9f30 con 0x7fb3180b4ff0 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:27.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.301+0000 7fb31f698640 10 monclient(hunting): authenticate will time out at 7369.406225s 2024-03-25T18:15:27.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.307+0000 7fb31cc33640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb3180b4670 0x7fb3181425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58866/0 (socket says 172.21.15.7:58866) 2024-03-25T18:15:27.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.307+0000 7fb31cc33640 1 -- 172.21.15.7:0/920101264 learned_addr learned my addr 172.21.15.7:0/920101264 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:27.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.316+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716595751 0 0) 0x7fb3180ab0e0 con 0x7fb3180b4670 2024-03-25T18:15:27.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.316+0000 7fb30ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:27.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.316+0000 7fb30ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:27.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.316+0000 7fb30ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:27.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.316+0000 7fb30ffff640 10 monclient(hunting): my global_id is 4272 2024-03-25T18:15:27.317 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.316+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb304003520 con 0x7fb3180b4670 2024-03-25T18:15:27.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 728645907 0 0) 0x7fb304003520 con 0x7fb3180b4670 2024-03-25T18:15:27.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 >> v1:172.21.15.7:6790/0 conn(0x7fb3180b4ff0 legacy=0x7fb31813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 >> v1:172.21.15.183:6789/0 conn(0x7fb31813c480 legacy=0x7fb31814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 1 monclient: found mon.noname-a 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3180a9b70 con 0x7fb3180b4670 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 10 monclient: _finish_auth 0 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb30ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb31f698640 5 monclient: authenticate success, global_id 4272 2024-03-25T18:15:27.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.323+0000 7fb31f698640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb308001690 con 0x7fb3180b4670 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 10 monclient: dump: 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:27.333 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb31f698640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb308001920 con 0x7fb3180b4670 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb31f698640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 1 -- 172.21.15.7:0/920101264 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb308001c30 con 0x7fb3180b4670 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb31f698640 4 set_mon_vals no callback set 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.331+0000 7fb30ffff640 10 monclient: dump: 2024-03-25T18:15:27.334 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.332+0000 7fb31f698640 10 monclient: shutdown 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.332+0000 7fb31f698640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.332+0000 7fb31f698640 1 -- 172.21.15.7:0/920101264 >> v1:172.21.15.7:6789/0 conn(0x7fb3180b4670 legacy=0x7fb3181425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.332+0000 7fb31f698640 1 -- 172.21.15.7:0/920101264 shutdown_connections 2024-03-25T18:15:27.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.332+0000 7fb31f698640 1 -- 172.21.15.7:0/920101264 >> 172.21.15.7:0/920101264 conn(0x7fb3180617a0 msgr2=0x7fb318061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:27.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.332+0000 7fb31f698640 1 -- 172.21.15.7:0/920101264 shutdown_connections 2024-03-25T18:15:27.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.332+0000 7fb31f698640 1 -- 172.21.15.7:0/920101264 wait complete. 2024-03-25T18:15:27.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 10 monclient: build_initial_monmap 2024-03-25T18:15:27.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 10 monclient: monmap: 2024-03-25T18:15:27.337 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:27.337 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:27.337 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 1 librados: starting msgr at 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 1 librados: starting objecter 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 1 Processor -- start 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 1 -- start start 2024-03-25T18:15:27.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 1 librados: setting wanted keys 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 1 librados: calling monclient init 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.333+0000 7fb31f698640 10 monclient: init 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 10 monclient(hunting): picked mon.noname-c con 0x7fb3180b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 10 monclient(hunting): picked mon.noname-a con 0x7fb3180b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 10 monclient(hunting): picked mon.noname-b con 0x7fb31813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3181452a0 con 0x7fb3180b4ff0 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb31813efe0 con 0x7fb31813c480 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb318144930 con 0x7fb3180b4670 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.334+0000 7fb31f698640 10 monclient(hunting): authenticate will time out at 7369.439336s 2024-03-25T18:15:27.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.345+0000 7fb31dc35640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb31813c480 0x7fb3181e2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:36026/0 (socket says 172.21.15.7:36026) 2024-03-25T18:15:27.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.345+0000 7fb31dc35640 1 -- 172.21.15.7:0/2428319028 learned_addr learned my addr 172.21.15.7:0/2428319028 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:27.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.347+0000 7fb30dffb640 10 client.?.objecter ms_handle_connect 0x7fb31813c480 2024-03-25T18:15:27.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.347+0000 7fb30dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:27.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.349+0000 7fb30dffb640 10 client.?.objecter ms_handle_connect 0x7fb3180b4670 2024-03-25T18:15:27.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.349+0000 7fb30dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.352+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1181399731 0 0) 0x7fb31813efe0 con 0x7fb31813c480 2024-03-25T18:15:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.352+0000 7fb30dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.352+0000 7fb30dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.352+0000 7fb30dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:27.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.352+0000 7fb30dffb640 10 monclient(hunting): my global_id is 4264 2024-03-25T18:15:27.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.353+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb2fc004eb0 con 0x7fb31813c480 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 10 client.?.objecter ms_handle_connect 0x7fb3180b4ff0 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 117566644 0 0) 0x7fb318144930 con 0x7fb3180b4670 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 10 monclient(hunting): my global_id is 4283 2024-03-25T18:15:27.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.355+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb31813efe0 con 0x7fb3180b4670 2024-03-25T18:15:27.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.362+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3095994173 0 0) 0x7fb2fc004eb0 con 0x7fb31813c480 2024-03-25T18:15:27.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.362+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb318144930 con 0x7fb31813c480 2024-03-25T18:15:27.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.362+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb310001c30 con 0x7fb31813c480 2024-03-25T18:15:27.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.362+0000 7fb30dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:27.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.362+0000 7fb30dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:27.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.362+0000 7fb30dffb640 10 monclient(hunting): dump: 2024-03-25T18:15:27.363 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:27.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.362+0000 7fb30dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:27.366 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.365+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 358665996 0 0) 0x7fb31813efe0 con 0x7fb3180b4670 2024-03-25T18:15:27.366 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.365+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb2fc004eb0 con 0x7fb3180b4670 2024-03-25T18:15:27.366 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.365+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb3140018a0 con 0x7fb3180b4670 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.365+0000 7fb30dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.365+0000 7fb30dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.365+0000 7fb30dffb640 10 monclient(hunting): dump: 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:27.367 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.372+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1469741390 0 0) 0x7fb318144930 con 0x7fb31813c480 2024-03-25T18:15:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 >> v1:172.21.15.7:6790/0 conn(0x7fb3180b4670 legacy=0x7fb318145710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 >> v1:172.21.15.7:6789/0 conn(0x7fb3180b4ff0 legacy=0x7fb318145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 1 monclient: found mon.b 2024-03-25T18:15:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb318144ba0 con 0x7fb31813c480 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb30dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 5 monclient: authenticate success, global_id 4264 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 10 monclient: _renew_subs 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb318144e80 con 0x7fb31813c480 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 10 client.4264.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 10 monclient: _renew_subs 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3181438d0 con 0x7fb31813c480 2024-03-25T18:15:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.373+0000 7fb31f698640 1 librados: init done 2024-03-25T18:15:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.374+0000 7fb31f698640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.374+0000 7fb31f698640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.374+0000 7fb31f698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.374+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb31813ee10 con 0x7fb31813c480 2024-03-25T18:15:27.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.644+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb310003990 con 0x7fb31813c480 2024-03-25T18:15:27.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.644+0000 7fb30dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:27.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.644+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb310003d20 con 0x7fb31813c480 2024-03-25T18:15:27.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.644+0000 7fb30dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.644+0000 7fb30dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.644+0000 7fb30dffb640 10 monclient: dump: 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:27.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.644+0000 7fb30f7fe640 4 set_mon_vals no callback set 2024-03-25T18:15:27.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.645+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fb310003f40 con 0x7fb31813c480 2024-03-25T18:15:27.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.646+0000 7fb31d434640 10 monclient: get_auth_request con 0x7fb2fc06de90 auth_method 0 2024-03-25T18:15:27.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.646+0000 7fb30dffb640 10 client.4264.objecter ms_handle_connect 0x7fb2fc06de90 2024-03-25T18:15:27.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fb310054a00 con 0x7fb31813c480 2024-03-25T18:15:27.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 10 client.4264.objecter ms_dispatch 0x7fb318000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:27.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 3 client.4264.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:27.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 3 client.4264.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:27.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 20 client.4264.objecter dump_active .. 0 homeless 2024-03-25T18:15:27.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb310050510 con 0x7fb31813c480 2024-03-25T18:15:27.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:27.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.650+0000 7fb30dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:27.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.873+0000 7fb31f698640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.873+0000 7fb31f698640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.873+0000 7fb31f698640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:27.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.873+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb31804eee0 con 0x7fb31813c480 2024-03-25T18:15:27.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.911+0000 7fb30dffb640 1 -- 172.21.15.7:0/2428319028 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fb310052440 con 0x7fb31813c480 2024-03-25T18:15:27.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.911+0000 7fb30dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:27.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.911+0000 7fb30dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:27.912 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:27.912 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:27.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 10 librados: watch_flush enter 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 10 librados: watch_flush exit 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 20 client.4264.objecter shutdown clearing up homeless session... 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 10 client.4264.objecter successfully canceled tick 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 >> v1:172.21.15.7:6800/1717564767 conn(0x7fb2fc06de90 legacy=0x7fb2fc070350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 10 monclient: shutdown 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 >> v1:172.21.15.183:6789/0 conn(0x7fb31813c480 legacy=0x7fb3181e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 shutdown_connections 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.913+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 >> 172.21.15.7:0/2428319028 conn(0x7fb3180617a0 msgr2=0x7fb3181407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:27.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.914+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 shutdown_connections 2024-03-25T18:15:27.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.914+0000 7fb31f698640 1 -- 172.21.15.7:0/2428319028 wait complete. 2024-03-25T18:15:27.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:27.914+0000 7fb31f698640 1 librados: shutdown 2024-03-25T18:15:27.926 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:28.116 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:34.753 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.128 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:34.765 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.182 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:34.809 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.184 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:34.821 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.189 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:34.826 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.192 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:34.829 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.192 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:34.819 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.239 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:34.877 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:28.245 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:34.872 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:34.732 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:34.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.849+0000 7f81e5df4640 10 monclient: get_monmap_and_config 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.849+0000 7f81e5df4640 10 monclient: build_initial_monmap 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.849+0000 7f81e5df4640 10 monclient: monmap: 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:34.850159+0000 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:34.850159+0000 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:34.850 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:34.851 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:34.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.849+0000 7f81e5df4640 1 Processor -- start 2024-03-25T18:15:34.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.849+0000 7f81e5df4640 1 -- start start 2024-03-25T18:15:34.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.849+0000 7f81e5df4640 10 monclient: init 2024-03-25T18:15:34.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.850+0000 7f81e5df4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:34.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.850+0000 7f81e5df4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:34.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.850+0000 7f81e5df4640 10 monclient(hunting): picked mon.noname-a con 0x7f81e00b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:34.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.850+0000 7f81e5df4640 10 monclient(hunting): picked mon.noname-b con 0x7f81e00b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:34.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.850+0000 7f81e5df4640 10 monclient(hunting): picked mon.noname-c con 0x7f81e013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:34.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.850+0000 7f81e5df4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00ab0e0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.851+0000 7f81e5df4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00ab3c0 con 0x7f81e013c480 2024-03-25T18:15:34.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.851+0000 7f81e5df4640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e00a9f30 con 0x7f81e00b4670 2024-03-25T18:15:34.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.851+0000 7f81e5df4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:34.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.851+0000 7f81e5df4640 10 monclient(hunting): authenticate will time out at 7376.955673s 2024-03-25T18:15:34.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.857+0000 7f81dffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f81e013c480 0x7f81e014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:37564/0 (socket says 172.21.15.7:37564) 2024-03-25T18:15:34.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.857+0000 7f81dffff640 1 -- 172.21.15.7:0/2635805486 learned_addr learned my addr 172.21.15.7:0/2635805486 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:34.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.870+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3659859324 0 0) 0x7f81e00ab3c0 con 0x7f81e013c480 2024-03-25T18:15:34.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.870+0000 7f81de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.870+0000 7f81de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.870+0000 7f81de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.870+0000 7f81de7fc640 10 monclient(hunting): my global_id is 4267 2024-03-25T18:15:34.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.870+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81c8003520 con 0x7f81e013c480 2024-03-25T18:15:34.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.877+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796842757 0 0) 0x7f81e00ab0e0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.877+0000 7f81de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.877+0000 7f81de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.877+0000 7f81de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.877+0000 7f81de7fc640 10 monclient(hunting): my global_id is 4278 2024-03-25T18:15:34.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.877+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e00ab3c0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2743681591 0 0) 0x7f81e00ab3c0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 >> v1:172.21.15.7:6790/0 conn(0x7f81e00b4670 legacy=0x7f81e01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 >> v1:172.21.15.183:6789/0 conn(0x7f81e013c480 legacy=0x7f81e014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 1 monclient: found mon.noname-a 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e00a9b70 con 0x7f81e00b4ff0 2024-03-25T18:15:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f81d0001670 con 0x7f81e00b4ff0 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81e5df4640 5 monclient: authenticate success, global_id 4278 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81e5df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81de7fc640 10 monclient: dump: 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:34.899 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:34.900 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:34.900 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:34.900 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:34.900 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.897+0000 7f81e5df4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d00017d0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81de7fc640 1 -- 172.21.15.7:0/2635805486 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f81d0001ba0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81de7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81de7fc640 10 monclient: dump: 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:34.903 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81e5df4640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:34.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.902+0000 7f81e5df4640 4 set_mon_vals no callback set 2024-03-25T18:15:34.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.903+0000 7f81e5df4640 10 monclient: shutdown 2024-03-25T18:15:34.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.903+0000 7f81e5df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:34.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.903+0000 7f81e5df4640 1 -- 172.21.15.7:0/2635805486 >> v1:172.21.15.7:6789/0 conn(0x7f81e00b4ff0 legacy=0x7f81e013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.903+0000 7f81e5df4640 1 -- 172.21.15.7:0/2635805486 shutdown_connections 2024-03-25T18:15:34.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.903+0000 7f81e5df4640 1 -- 172.21.15.7:0/2635805486 >> 172.21.15.7:0/2635805486 conn(0x7f81e00617a0 msgr2=0x7f81e0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.903+0000 7f81e5df4640 1 -- 172.21.15.7:0/2635805486 shutdown_connections 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.903+0000 7f81e5df4640 1 -- 172.21.15.7:0/2635805486 wait complete. 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.904+0000 7f81e5df4640 10 monclient: build_initial_monmap 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.904+0000 7f81e5df4640 10 monclient: monmap: 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.904+0000 7f81e5df4640 1 librados: starting msgr at 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.904+0000 7f81e5df4640 1 librados: starting objecter 2024-03-25T18:15:34.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.904+0000 7f81e5df4640 1 Processor -- start 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 1 -- start start 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 1 librados: setting wanted keys 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 1 librados: calling monclient init 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient: init 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient(hunting): picked mon.noname-b con 0x7f81e00b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient(hunting): picked mon.noname-c con 0x7f81e00b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient(hunting): picked mon.noname-a con 0x7f81e013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0143880 con 0x7f81e013c480 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0144890 con 0x7f81e00b4670 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81e0144ae0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:34.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.905+0000 7f81e5df4640 10 monclient(hunting): authenticate will time out at 7377.010302s 2024-03-25T18:15:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.918+0000 7f81deffd640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f81e00b4ff0 0x7f81e0145f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:38132/0 (socket says 172.21.15.7:38132) 2024-03-25T18:15:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.918+0000 7f81deffd640 1 -- 172.21.15.7:0/478114455 learned_addr learned my addr 172.21.15.7:0/478114455 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:34.920 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.919+0000 7f81c7fff640 10 client.?.objecter ms_handle_connect 0x7f81e00b4ff0 2024-03-25T18:15:34.920 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.919+0000 7f81c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.922 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.922+0000 7f81c7fff640 10 client.?.objecter ms_handle_connect 0x7f81e013c480 2024-03-25T18:15:34.922 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.922+0000 7f81c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.923+0000 7f81c7fff640 10 client.?.objecter ms_handle_connect 0x7f81e00b4670 2024-03-25T18:15:34.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.923+0000 7f81c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.927+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2181745546 0 0) 0x7f81e0144ae0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.927+0000 7f81c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.927+0000 7f81c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.927+0000 7f81c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.927+0000 7f81c7fff640 10 monclient(hunting): my global_id is 4289 2024-03-25T18:15:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.927+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc004eb0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.935+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2211033842 0 0) 0x7f81e0143880 con 0x7f81e013c480 2024-03-25T18:15:34.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.935+0000 7f81c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.935+0000 7f81c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.935+0000 7f81c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.935+0000 7f81c7fff640 10 monclient(hunting): my global_id is 4281 2024-03-25T18:15:34.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.935+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81e0144ae0 con 0x7f81e013c480 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3553439427 0 0) 0x7f81cc004eb0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81e0143880 con 0x7f81e00b4ff0 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f81d4001a30 con 0x7f81e00b4ff0 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): dump: 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:34.941 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2674975348 0 0) 0x7f81e0144890 con 0x7f81e00b4670 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 10 monclient(hunting): my global_id is 4270 2024-03-25T18:15:34.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.940+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc004eb0 con 0x7f81e00b4670 2024-03-25T18:15:34.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4100597056 0 0) 0x7f81e0143880 con 0x7f81e00b4ff0 2024-03-25T18:15:34.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 >> v1:172.21.15.183:6789/0 conn(0x7f81e00b4670 legacy=0x7f81e0145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 >> v1:172.21.15.7:6789/0 conn(0x7f81e013c480 legacy=0x7f81e01e2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:34.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:34.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 1 monclient: found mon.c 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81e0144dc0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 5 monclient: authenticate success, global_id 4289 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 10 monclient: _renew_subs 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81e013f1a0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 10 client.4289.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 10 monclient: _renew_subs 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81e013dbe0 con 0x7f81e00b4ff0 2024-03-25T18:15:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.947+0000 7f81e5df4640 1 librados: init done 2024-03-25T18:15:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.948+0000 7f81e5df4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.948+0000 7f81e5df4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.948+0000 7f81e5df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:34.948+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81e013dee0 con 0x7f81e00b4ff0 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.265+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81d4001c70 con 0x7f81e00b4ff0 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.265+0000 7f81c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.265+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f81d4003a60 con 0x7f81e00b4ff0 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.265+0000 7f81c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.265+0000 7f81c7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.265+0000 7f81c7fff640 10 monclient: dump: 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:35.266 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:35.267 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:35.267 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:35.267 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:35.267 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:35.267 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:35.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.265+0000 7f81ddffb640 4 set_mon_vals no callback set 2024-03-25T18:15:35.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.266+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f81d4003c80 con 0x7f81e00b4ff0 2024-03-25T18:15:35.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.266+0000 7f81df7fe640 10 monclient: get_auth_request con 0x7f81cc06de00 auth_method 0 2024-03-25T18:15:35.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.266+0000 7f81c7fff640 10 client.4289.objecter ms_handle_connect 0x7f81cc06de00 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f81d40548a0 con 0x7f81e00b4ff0 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 10 client.4289.objecter ms_dispatch 0x7f81e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 3 client.4289.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 3 client.4289.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 20 client.4289.objecter dump_active .. 0 homeless 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81d40503b0 con 0x7f81e00b4ff0 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:35.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.273+0000 7f81c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:35.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.523+0000 7f81e5df4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:35.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.523+0000 7f81e5df4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:35.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.523+0000 7f81e5df4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:15:35.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.523+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81e006dbd0 con 0x7f81e00b4ff0 2024-03-25T18:15:35.545 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.544+0000 7f81c7fff640 1 -- 172.21.15.7:0/478114455 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f81d40522e0 con 0x7f81e00b4ff0 2024-03-25T18:15:35.545 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.544+0000 7f81c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:35.545 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.544+0000 7f81c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:35.545 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:35.546 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 10 librados: watch_flush enter 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 10 librados: watch_flush exit 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 20 client.4289.objecter shutdown clearing up homeless session... 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 10 client.4289.objecter successfully canceled tick 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 >> v1:172.21.15.7:6800/1717564767 conn(0x7f81cc06de00 legacy=0x7f81cc0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 10 monclient: shutdown 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 >> v1:172.21.15.7:6790/0 conn(0x7f81e00b4ff0 legacy=0x7f81e0145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 shutdown_connections 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 >> 172.21.15.7:0/478114455 conn(0x7f81e00617a0 msgr2=0x7f81e0140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 shutdown_connections 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.546+0000 7f81e5df4640 1 -- 172.21.15.7:0/478114455 wait complete. 2024-03-25T18:15:35.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:35.547+0000 7f81e5df4640 1 librados: shutdown 2024-03-25T18:15:35.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:42.367 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.481+0000 7fe31b896640 10 monclient: get_monmap_and_config 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.481+0000 7fe31b896640 10 monclient: build_initial_monmap 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.481+0000 7fe31b896640 10 monclient: monmap: 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:42.482405+0000 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:42.482405+0000 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:42.482 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:42.483 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.481+0000 7fe31b896640 1 Processor -- start 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 1 -- start start 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient: init 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient(hunting): picked mon.noname-a con 0x7fe3140b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient(hunting): picked mon.noname-b con 0x7fe3140b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient(hunting): picked mon.noname-c con 0x7fe3140b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140a96f0 con 0x7fe3140b4e70 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3140ab0e0 con 0x7fe3140b1b00 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe314145f70 con 0x7fe3140b44b0 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:42.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.482+0000 7fe31b896640 10 monclient(hunting): authenticate will time out at 7384.587592s 2024-03-25T18:15:42.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.489+0000 7fe318e31640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fe3140b44b0 0x7fe314145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:44926/0 (socket says 172.21.15.7:44926) 2024-03-25T18:15:42.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.489+0000 7fe318e31640 1 -- 172.21.15.7:0/2825173536 learned_addr learned my addr 172.21.15.7:0/2825173536 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:42.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.502+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2292353837 0 0) 0x7fe314145f70 con 0x7fe3140b44b0 2024-03-25T18:15:42.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.502+0000 7fe30bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.502+0000 7fe30bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.502+0000 7fe30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.502+0000 7fe30bfff640 10 monclient(hunting): my global_id is 4292 2024-03-25T18:15:42.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.502+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe300003520 con 0x7fe3140b44b0 2024-03-25T18:15:42.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.506+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403812533 0 0) 0x7fe3140a96f0 con 0x7fe3140b4e70 2024-03-25T18:15:42.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.506+0000 7fe30bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.506+0000 7fe30bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.506+0000 7fe30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.506+0000 7fe30bfff640 10 monclient(hunting): my global_id is 4284 2024-03-25T18:15:42.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.506+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe314145f70 con 0x7fe3140b4e70 2024-03-25T18:15:42.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.514+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3500357361 0 0) 0x7fe3140ab0e0 con 0x7fe3140b1b00 2024-03-25T18:15:42.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.514+0000 7fe30bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.514+0000 7fe30bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.514+0000 7fe30bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.514+0000 7fe30bfff640 10 monclient(hunting): my global_id is 4273 2024-03-25T18:15:42.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.514+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3140a96f0 con 0x7fe3140b1b00 2024-03-25T18:15:42.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2070190881 0 0) 0x7fe314145f70 con 0x7fe3140b4e70 2024-03-25T18:15:42.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 >> v1:172.21.15.7:6790/0 conn(0x7fe3140b44b0 legacy=0x7fe314145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 >> v1:172.21.15.183:6789/0 conn(0x7fe3140b1b00 legacy=0x7fe31414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:42.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 1 monclient: found mon.noname-a 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3140a9b80 con 0x7fe3140b4e70 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 10 monclient: _finish_auth 0 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe310001670 con 0x7fe3140b4e70 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe31b896640 5 monclient: authenticate success, global_id 4284 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe31b896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe30bfff640 10 monclient: dump: 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.517 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.518 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.518 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:42.518 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:42.518 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:42.518 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:42.518 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:42.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.515+0000 7fe31b896640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe3100017d0 con 0x7fe3140b4e70 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe30bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe30bfff640 1 -- 172.21.15.7:0/2825173536 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe310001be0 con 0x7fe3140b4e70 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe31b896640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe31b896640 4 set_mon_vals no callback set 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe30bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe30bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.523+0000 7fe30bfff640 10 monclient: dump: 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.524 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:42.525 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:42.525 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:42.525 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:42.525 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:42.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.524+0000 7fe31b896640 10 monclient: shutdown 2024-03-25T18:15:42.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.524+0000 7fe31b896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:42.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.524+0000 7fe31b896640 1 -- 172.21.15.7:0/2825173536 >> v1:172.21.15.7:6789/0 conn(0x7fe3140b4e70 legacy=0x7fe31413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.524+0000 7fe31b896640 1 -- 172.21.15.7:0/2825173536 shutdown_connections 2024-03-25T18:15:42.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.524+0000 7fe31b896640 1 -- 172.21.15.7:0/2825173536 >> 172.21.15.7:0/2825173536 conn(0x7fe3140617a0 msgr2=0x7fe314061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.524+0000 7fe31b896640 1 -- 172.21.15.7:0/2825173536 shutdown_connections 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.524+0000 7fe31b896640 1 -- 172.21.15.7:0/2825173536 wait complete. 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.525+0000 7fe31b896640 10 monclient: build_initial_monmap 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.525+0000 7fe31b896640 10 monclient: monmap: 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:42.527 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:42.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.525+0000 7fe31b896640 1 librados: starting msgr at 2024-03-25T18:15:42.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.525+0000 7fe31b896640 1 librados: starting objecter 2024-03-25T18:15:42.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.525+0000 7fe31b896640 1 Processor -- start 2024-03-25T18:15:42.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 1 -- start start 2024-03-25T18:15:42.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 1 librados: setting wanted keys 2024-03-25T18:15:42.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 1 librados: calling monclient init 2024-03-25T18:15:42.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 10 monclient: init 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 10 monclient(hunting): picked mon.noname-c con 0x7fe3140b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 10 monclient(hunting): picked mon.noname-a con 0x7fe3140b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 10 monclient(hunting): picked mon.noname-b con 0x7fe3140b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3141d7750 con 0x7fe3140b44b0 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.526+0000 7fe31b896640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3141d8390 con 0x7fe3140b4e70 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.527+0000 7fe31b896640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3141d8520 con 0x7fe3140b1b00 2024-03-25T18:15:42.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.527+0000 7fe31b896640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:42.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.527+0000 7fe31b896640 10 monclient(hunting): authenticate will time out at 7384.631662s 2024-03-25T18:15:42.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.535+0000 7fe319e33640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe3140b4e70 0x7fe3141daff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:33268/0 (socket says 172.21.15.7:33268) 2024-03-25T18:15:42.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.535+0000 7fe319e33640 1 -- 172.21.15.7:0/3418968927 learned_addr learned my addr 172.21.15.7:0/3418968927 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:42.540 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.539+0000 7fe309ffb640 10 client.?.objecter ms_handle_connect 0x7fe3140b4e70 2024-03-25T18:15:42.540 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.539+0000 7fe309ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:42.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.547+0000 7fe309ffb640 10 client.?.objecter ms_handle_connect 0x7fe3140b1b00 2024-03-25T18:15:42.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.547+0000 7fe309ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:42.552 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.551+0000 7fe309ffb640 10 client.?.objecter ms_handle_connect 0x7fe3140b44b0 2024-03-25T18:15:42.552 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.551+0000 7fe309ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:42.552 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.551+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3403658029 0 0) 0x7fe3141d8390 con 0x7fe3140b4e70 2024-03-25T18:15:42.552 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.552+0000 7fe309ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.552 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.552+0000 7fe309ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.552 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.552+0000 7fe309ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.552+0000 7fe309ffb640 10 monclient(hunting): my global_id is 4276 2024-03-25T18:15:42.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.552+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f8004eb0 con 0x7fe3140b4e70 2024-03-25T18:15:42.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.560+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2821638616 0 0) 0x7fe3141d8520 con 0x7fe3140b1b00 2024-03-25T18:15:42.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.560+0000 7fe309ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.560+0000 7fe309ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.560+0000 7fe309ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.560+0000 7fe309ffb640 10 monclient(hunting): my global_id is 4295 2024-03-25T18:15:42.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.560+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3141d8390 con 0x7fe3140b1b00 2024-03-25T18:15:42.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.565+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 770623791 0 0) 0x7fe2f8004eb0 con 0x7fe3140b4e70 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.565+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe3141d8520 con 0x7fe3140b4e70 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.565+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe30c001d10 con 0x7fe3140b4e70 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.565+0000 7fe309ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.565+0000 7fe309ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.565+0000 7fe309ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:42.566 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:42.567 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:42.567 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:42.567 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:42.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.565+0000 7fe309ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:42.576 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.575+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220544618 0 0) 0x7fe3141d7750 con 0x7fe3140b44b0 2024-03-25T18:15:42.576 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.575+0000 7fe309ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:42.576 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.575+0000 7fe309ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:42.576 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.575+0000 7fe309ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:42.576 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.575+0000 7fe309ffb640 10 monclient(hunting): my global_id is 4287 2024-03-25T18:15:42.576 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.575+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe2f8004eb0 con 0x7fe3140b44b0 2024-03-25T18:15:42.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1994119070 0 0) 0x7fe3141d8520 con 0x7fe3140b4e70 2024-03-25T18:15:42.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 >> v1:172.21.15.7:6790/0 conn(0x7fe3140b1b00 legacy=0x7fe3141d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 >> v1:172.21.15.7:6789/0 conn(0x7fe3140b44b0 legacy=0x7fe31413aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:42.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:42.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 1 monclient: found mon.b 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3141d86b0 con 0x7fe3140b4e70 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe309ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe31b896640 5 monclient: authenticate success, global_id 4276 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe31b896640 10 monclient: _renew_subs 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe31b896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.578+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe314145470 con 0x7fe3140b4e70 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.579+0000 7fe31b896640 10 client.4276.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.579+0000 7fe31b896640 10 monclient: _renew_subs 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.579+0000 7fe31b896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:42.580 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.579+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe31413b790 con 0x7fe3140b4e70 2024-03-25T18:15:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.579+0000 7fe31b896640 1 librados: init done 2024-03-25T18:15:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.580+0000 7fe31b896640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.580+0000 7fe31b896640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.580+0000 7fe31b896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.580+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3141dae60 con 0x7fe3140b4e70 2024-03-25T18:15:42.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.794+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe30c003990 con 0x7fe3140b4e70 2024-03-25T18:15:42.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.794+0000 7fe309ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:42.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.794+0000 7fe30b7fe640 4 set_mon_vals no callback set 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.852+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe30c003d60 con 0x7fe3140b4e70 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.852+0000 7fe309ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.852+0000 7fe309ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.852+0000 7fe309ffb640 10 monclient: dump: 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:42.853 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:42.854 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:42.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.853+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fe30c003f40 con 0x7fe3140b4e70 2024-03-25T18:15:42.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.853+0000 7fe319632640 10 monclient: get_auth_request con 0x7fe2f806de00 auth_method 0 2024-03-25T18:15:42.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.853+0000 7fe309ffb640 10 client.4276.objecter ms_handle_connect 0x7fe2f806de00 2024-03-25T18:15:42.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.855+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fe30c054b40 con 0x7fe3140b4e70 2024-03-25T18:15:42.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.855+0000 7fe309ffb640 10 client.4276.objecter ms_dispatch 0x7fe314000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:42.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.855+0000 7fe309ffb640 3 client.4276.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:42.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.855+0000 7fe309ffb640 3 client.4276.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:42.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.855+0000 7fe309ffb640 20 client.4276.objecter dump_active .. 0 homeless 2024-03-25T18:15:42.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.858+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe30c050480 con 0x7fe3140b4e70 2024-03-25T18:15:42.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.858+0000 7fe309ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:42.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:42.858+0000 7fe309ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:43.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.077+0000 7fe31b896640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:43.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.077+0000 7fe31b896640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:43.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.077+0000 7fe31b896640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:43.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.077+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3141dac50 con 0x7fe3140b4e70 2024-03-25T18:15:43.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.105+0000 7fe309ffb640 1 -- 172.21.15.7:0/3418968927 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fe30c052580 con 0x7fe3140b4e70 2024-03-25T18:15:43.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.105+0000 7fe309ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:43.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.105+0000 7fe309ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:43.106 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:43.106 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:43.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.107+0000 7fe31b896640 10 librados: watch_flush enter 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.107+0000 7fe31b896640 10 librados: watch_flush exit 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.107+0000 7fe31b896640 20 client.4276.objecter shutdown clearing up homeless session... 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.107+0000 7fe31b896640 10 client.4276.objecter successfully canceled tick 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 >> v1:172.21.15.7:6800/1717564767 conn(0x7fe2f806de00 legacy=0x7fe2f80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 10 monclient: shutdown 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 >> v1:172.21.15.183:6789/0 conn(0x7fe3140b4e70 legacy=0x7fe3141daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 shutdown_connections 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 >> 172.21.15.7:0/3418968927 conn(0x7fe3140617a0 msgr2=0x7fe3140b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 shutdown_connections 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 1 -- 172.21.15.7:0/3418968927 wait complete. 2024-03-25T18:15:43.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:43.108+0000 7fe31b896640 1 librados: shutdown 2024-03-25T18:15:43.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:48.189 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:54.826 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.195 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:54.832 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.223 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:01:54.860 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.241 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:54.868 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.247 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:54.874 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.253 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:54.890 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.261 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:54.899 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.287 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:01:54.914 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:48.294 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:01:54.932 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:15:49.926 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:50.041 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.040+0000 7f4ff73b5640 10 monclient: get_monmap_and_config 2024-03-25T18:15:50.041 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.040+0000 7f4ff73b5640 10 monclient: build_initial_monmap 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.040+0000 7f4ff73b5640 10 monclient: monmap: 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:50.041793+0000 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:50.041793+0000 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:50.042 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:50.043 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:50.043 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.041+0000 7f4ff73b5640 1 Processor -- start 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.041+0000 7f4ff73b5640 1 -- start start 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.041+0000 7f4ff73b5640 10 monclient: init 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff00b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 10 monclient(hunting): picked mon.noname-a con 0x7f4ff00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00ab0e0 con 0x7f4ff00b4670 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00ab3c0 con 0x7f4ff013c480 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00a9f30 con 0x7f4ff00b4ff0 2024-03-25T18:15:50.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:50.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.042+0000 7f4ff73b5640 10 monclient(hunting): authenticate will time out at 7392.147315s 2024-03-25T18:15:50.047 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.047+0000 7f4ff4950640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4ff00b4670 0x7f4ff01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50026/0 (socket says 172.21.15.7:50026) 2024-03-25T18:15:50.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.047+0000 7f4ff4950640 1 -- 172.21.15.7:0/3496313274 learned_addr learned my addr 172.21.15.7:0/3496313274 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:50.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.051+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4233039979 0 0) 0x7f4ff00ab0e0 con 0x7f4ff00b4670 2024-03-25T18:15:50.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.051+0000 7f4fe7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:50.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.051+0000 7f4fe7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:50.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.051+0000 7f4fe7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:50.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.051+0000 7f4fe7fff640 10 monclient(hunting): my global_id is 4290 2024-03-25T18:15:50.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.051+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fdc003520 con 0x7f4ff00b4670 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4156622348 0 0) 0x7f4fdc003520 con 0x7f4ff00b4670 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 >> v1:172.21.15.7:6790/0 conn(0x7f4ff00b4ff0 legacy=0x7f4ff013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 >> v1:172.21.15.183:6789/0 conn(0x7f4ff013c480 legacy=0x7f4ff014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 1 monclient: found mon.noname-a 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff00a9b70 con 0x7f4ff00b4670 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 10 monclient: _finish_auth 0 2024-03-25T18:15:50.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4fe0001690 con 0x7f4ff00b4670 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4fe7fff640 10 monclient: dump: 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:50.061 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:50.062 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:50.062 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:50.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4ff73b5640 5 monclient: authenticate success, global_id 4290 2024-03-25T18:15:50.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.059+0000 7f4ff73b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:50.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.062+0000 7f4fe7fff640 1 -- 172.21.15.7:0/3496313274 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fe00017f0 con 0x7f4ff00b4670 2024-03-25T18:15:50.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.062+0000 7f4fe7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:50.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.062+0000 7f4ff73b5640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:50.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.062+0000 7f4ff73b5640 4 set_mon_vals no callback set 2024-03-25T18:15:50.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.063+0000 7f4ff73b5640 10 monclient: shutdown 2024-03-25T18:15:50.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.063+0000 7f4ff73b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:50.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.063+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3496313274 >> v1:172.21.15.7:6789/0 conn(0x7f4ff00b4670 legacy=0x7f4ff01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.063+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3496313274 shutdown_connections 2024-03-25T18:15:50.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.063+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3496313274 >> 172.21.15.7:0/3496313274 conn(0x7f4ff00617a0 msgr2=0x7f4ff0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:50.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.063+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3496313274 shutdown_connections 2024-03-25T18:15:50.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.063+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3496313274 wait complete. 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient: build_initial_monmap 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient: monmap: 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:50.066 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 librados: starting msgr at 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 librados: starting objecter 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 Processor -- start 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 -- start start 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 librados: setting wanted keys 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 librados: calling monclient init 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient: init 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient(hunting): picked mon.noname-a con 0x7f4ff00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:50.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient(hunting): picked mon.noname-c con 0x7f4ff00b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:50.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 10 monclient(hunting): picked mon.noname-b con 0x7f4ff013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:50.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00b0a30 con 0x7f4ff00b4670 2024-03-25T18:15:50.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.064+0000 7f4ff73b5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff01d84a0 con 0x7f4ff013c480 2024-03-25T18:15:50.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.065+0000 7f4ff73b5640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ff00b0c80 con 0x7f4ff00b4ff0 2024-03-25T18:15:50.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.065+0000 7f4ff73b5640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:50.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.065+0000 7f4ff73b5640 10 monclient(hunting): authenticate will time out at 7392.169655s 2024-03-25T18:15:50.071 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.070+0000 7f4ff5151640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4ff00b4670 0x7f4ff00ace60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:50068/0 (socket says 172.21.15.7:50068) 2024-03-25T18:15:50.071 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.070+0000 7f4ff5151640 1 -- 172.21.15.7:0/3449512078 learned_addr learned my addr 172.21.15.7:0/3449512078 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:50.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.071+0000 7f4fe5ffb640 10 client.?.objecter ms_handle_connect 0x7f4ff00b4670 2024-03-25T18:15:50.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.071+0000 7f4fe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 10 client.?.objecter ms_handle_connect 0x7f4ff013c480 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046550035 0 0) 0x7f4ff00b0a30 con 0x7f4ff00b4670 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 10 monclient(hunting): my global_id is 4293 2024-03-25T18:15:50.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.082+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd4004eb0 con 0x7f4ff00b4670 2024-03-25T18:15:50.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.092+0000 7f4fe5ffb640 10 client.?.objecter ms_handle_connect 0x7f4ff00b4ff0 2024-03-25T18:15:50.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.092+0000 7f4fe5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:50.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.095+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4060835241 0 0) 0x7f4fd4004eb0 con 0x7f4ff00b4670 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.095+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ff00b0a30 con 0x7f4ff00b4670 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.095+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4fec001a30 con 0x7f4ff00b4670 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.095+0000 7f4fe5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.095+0000 7f4fe5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.095+0000 7f4fe5ffb640 10 monclient(hunting): dump: 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:50.096 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:50.097 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:50.097 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:50.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.095+0000 7f4fe5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:50.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.103+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4288930662 0 0) 0x7f4ff00b0c80 con 0x7f4ff00b4ff0 2024-03-25T18:15:50.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.103+0000 7f4fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:50.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.103+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:50.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.103+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:50.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.103+0000 7f4fe5ffb640 10 monclient(hunting): my global_id is 4301 2024-03-25T18:15:50.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.103+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4fd4004eb0 con 0x7f4ff00b4ff0 2024-03-25T18:15:50.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.104+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4134461498 0 0) 0x7f4ff01d84a0 con 0x7f4ff013c480 2024-03-25T18:15:50.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.104+0000 7f4fe5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:15:50.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.104+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:50.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.104+0000 7f4fe5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:50.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.104+0000 7f4fe5ffb640 10 monclient(hunting): my global_id is 4282 2024-03-25T18:15:50.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.104+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ff00b0c80 con 0x7f4ff013c480 2024-03-25T18:15:50.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2957863393 0 0) 0x7f4ff00b0a30 con 0x7f4ff00b4670 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 >> v1:172.21.15.7:6790/0 conn(0x7f4ff00b4ff0 legacy=0x7f4ff00b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 >> v1:172.21.15.183:6789/0 conn(0x7f4ff013c480 legacy=0x7f4ff01d40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 1 monclient: found mon.a 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff01d7f80 con 0x7f4ff00b4670 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4fe5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:50.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4ff73b5640 5 monclient: authenticate success, global_id 4293 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4ff73b5640 10 monclient: _renew_subs 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4ff73b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.107+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ff01d51d0 con 0x7f4ff00b4670 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 10 client.4293.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 10 monclient: _renew_subs 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ff01d7d30 con 0x7f4ff00b4670 2024-03-25T18:15:50.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 1 librados: init done 2024-03-25T18:15:50.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:50.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:50.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:50.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.108+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ff006dbd0 con 0x7f4ff00b4670 2024-03-25T18:15:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.381+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4fec001cb0 con 0x7f4ff00b4670 2024-03-25T18:15:50.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.381+0000 7f4fe5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:50.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.381+0000 7f4fe77fe640 4 set_mon_vals no callback set 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.422+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4fec003a60 con 0x7f4ff00b4670 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.422+0000 7f4fe5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.422+0000 7f4fe5ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.422+0000 7f4fe5ffb640 10 monclient: dump: 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:50.423 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:50.424 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:50.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.428+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f4fec003c40 con 0x7f4ff00b4670 2024-03-25T18:15:50.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.428+0000 7f4ff4950640 10 monclient: get_auth_request con 0x7f4fd406de80 auth_method 0 2024-03-25T18:15:50.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.429+0000 7f4fe5ffb640 10 client.4293.objecter ms_handle_connect 0x7f4fd406de80 2024-03-25T18:15:50.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.434+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f4fec054960 con 0x7f4ff00b4670 2024-03-25T18:15:50.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.434+0000 7f4fe5ffb640 10 client.4293.objecter ms_dispatch 0x7f4ff0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:50.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.434+0000 7f4fe5ffb640 3 client.4293.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:50.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.434+0000 7f4fe5ffb640 3 client.4293.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:50.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.434+0000 7f4fe5ffb640 20 client.4293.objecter dump_active .. 0 homeless 2024-03-25T18:15:50.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.435+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4fec0502a0 con 0x7f4ff00b4670 2024-03-25T18:15:50.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.435+0000 7f4fe5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:50.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.435+0000 7f4fe5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:50.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.676+0000 7f4ff73b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:50.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.676+0000 7f4ff73b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:50.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.676+0000 7f4ff73b5640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:15:50.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.676+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ff00aa590 con 0x7f4ff00b4670 2024-03-25T18:15:50.703 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.702+0000 7f4fe5ffb640 1 -- 172.21.15.7:0/3449512078 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f4fec0523a0 con 0x7f4ff00b4670 2024-03-25T18:15:50.703 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.702+0000 7f4fe5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:50.703 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.702+0000 7f4fe5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:50.704 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:50.704 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:50.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 10 librados: watch_flush enter 2024-03-25T18:15:50.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 10 librados: watch_flush exit 2024-03-25T18:15:50.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 20 client.4293.objecter shutdown clearing up homeless session... 2024-03-25T18:15:50.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 10 client.4293.objecter successfully canceled tick 2024-03-25T18:15:50.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 >> v1:172.21.15.7:6800/1717564767 conn(0x7f4fd406de80 legacy=0x7f4fd4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 10 monclient: shutdown 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 >> v1:172.21.15.7:6789/0 conn(0x7f4ff00b4670 legacy=0x7f4ff00ace60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 shutdown_connections 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 >> 172.21.15.7:0/3449512078 conn(0x7f4ff00617a0 msgr2=0x7f4ff0145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 shutdown_connections 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 1 -- 172.21.15.7:0/3449512078 wait complete. 2024-03-25T18:15:50.707 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:50.705+0000 7f4ff73b5640 1 librados: shutdown 2024-03-25T18:15:50.718 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:15:57.525 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:15:57.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.639+0000 7f1eedd04640 10 monclient: get_monmap_and_config 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.639+0000 7f1eedd04640 10 monclient: build_initial_monmap 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.639+0000 7f1eedd04640 10 monclient: monmap: 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:15:57.640287+0000 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:15:57.640287+0000 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:15:57.640 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:57.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.640+0000 7f1eedd04640 1 Processor -- start 2024-03-25T18:15:57.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.640+0000 7f1eedd04640 1 -- start start 2024-03-25T18:15:57.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.640+0000 7f1eedd04640 10 monclient: init 2024-03-25T18:15:57.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee80b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 10 monclient(hunting): picked mon.noname-a con 0x7f1ee80b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 10 monclient(hunting): picked mon.noname-c con 0x7f1ee80b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee80a96f0 con 0x7f1ee80b44b0 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee80ab0e0 con 0x7f1ee80b1b00 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee8145f70 con 0x7f1ee80b4e70 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:57.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.641+0000 7f1eedd04640 10 monclient(hunting): authenticate will time out at 7399.745842s 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.648+0000 7f1ee77fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f1ee80b4e70 0x7f1ee813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:47100/0 (socket says 172.21.15.7:47100) 2024-03-25T18:15:57.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.649+0000 7f1ee77fe640 1 -- 172.21.15.7:0/2891736566 learned_addr learned my addr 172.21.15.7:0/2891736566 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:57.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.662+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1405775026 0 0) 0x7f1ee80ab0e0 con 0x7f1ee80b1b00 2024-03-25T18:15:57.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.662+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.662+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.662+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.662+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 4285 2024-03-25T18:15:57.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.663+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed0003520 con 0x7f1ee80b1b00 2024-03-25T18:15:57.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.669+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 291146274 0 0) 0x7f1ee80a96f0 con 0x7f1ee80b44b0 2024-03-25T18:15:57.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.669+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.669+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.669+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.669+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 4296 2024-03-25T18:15:57.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.669+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee80ab0e0 con 0x7f1ee80b44b0 2024-03-25T18:15:57.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.671+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3166047217 0 0) 0x7f1ee8145f70 con 0x7f1ee80b4e70 2024-03-25T18:15:57.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.671+0000 7f1ee67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.671+0000 7f1ee67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.671+0000 7f1ee67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.671+0000 7f1ee67fc640 10 monclient(hunting): my global_id is 4304 2024-03-25T18:15:57.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.671+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee80a96f0 con 0x7f1ee80b4e70 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 51973428 0 0) 0x7f1ed0003520 con 0x7f1ee80b1b00 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 >> v1:172.21.15.7:6790/0 conn(0x7f1ee80b4e70 legacy=0x7f1ee813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 >> v1:172.21.15.7:6789/0 conn(0x7f1ee80b44b0 legacy=0x7f1ee8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 1 monclient: found mon.noname-c 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:15:57.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee80a9b80 con 0x7f1ee80b1b00 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 10 monclient: _finish_auth 0 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1ee0001a10 con 0x7f1ee80b1b00 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1ee67fc640 10 monclient: dump: 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.678 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.679 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:57.679 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:57.679 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:57.679 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:57.679 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:57.679 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1eedd04640 5 monclient: authenticate success, global_id 4285 2024-03-25T18:15:57.679 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.676+0000 7f1eedd04640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ee0001b70 con 0x7f1ee80b1b00 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1ee67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1ee67fc640 1 -- 172.21.15.7:0/2891736566 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1ee0001f80 con 0x7f1ee80b1b00 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1eedd04640 10 monclient: get_monmap_and_config success 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1eedd04640 4 set_mon_vals no callback set 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1ee67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1ee67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.690+0000 7f1ee67fc640 10 monclient: dump: 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.691 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.692 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:57.692 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:57.692 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:57.692 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:57.692 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:57.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.691+0000 7f1eedd04640 10 monclient: shutdown 2024-03-25T18:15:57.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.691+0000 7f1eedd04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:57.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.691+0000 7f1eedd04640 1 -- 172.21.15.7:0/2891736566 >> v1:172.21.15.183:6789/0 conn(0x7f1ee80b1b00 legacy=0x7f1ee814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.691+0000 7f1eedd04640 1 -- 172.21.15.7:0/2891736566 shutdown_connections 2024-03-25T18:15:57.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.691+0000 7f1eedd04640 1 -- 172.21.15.7:0/2891736566 >> 172.21.15.7:0/2891736566 conn(0x7f1ee80617a0 msgr2=0x7f1ee8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:57.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.691+0000 7f1eedd04640 1 -- 172.21.15.7:0/2891736566 shutdown_connections 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.691+0000 7f1eedd04640 1 -- 172.21.15.7:0/2891736566 wait complete. 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 10 monclient: build_initial_monmap 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 10 monclient: monmap: 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 1 librados: starting msgr at 2024-03-25T18:15:57.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 1 librados: starting objecter 2024-03-25T18:15:57.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 1 Processor -- start 2024-03-25T18:15:57.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 1 -- start start 2024-03-25T18:15:57.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 1 librados: setting wanted keys 2024-03-25T18:15:57.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 1 librados: calling monclient init 2024-03-25T18:15:57.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.692+0000 7f1eedd04640 10 monclient: init 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 10 monclient: _reopen_session rank -1 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 10 monclient(hunting): picked mon.noname-b con 0x7f1ee80b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 10 monclient(hunting): picked mon.noname-c con 0x7f1ee80b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 10 monclient(hunting): picked mon.noname-a con 0x7f1ee80b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee8143600 con 0x7f1ee80b4e70 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee8140c50 con 0x7f1ee80b1b00 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ee813cc00 con 0x7f1ee80b44b0 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 10 monclient(hunting): _renew_subs 2024-03-25T18:15:57.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.693+0000 7f1eedd04640 10 monclient(hunting): authenticate will time out at 7399.798467s 2024-03-25T18:15:57.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.701+0000 7f1ee77fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1ee80b1b00 0x7f1ee8140540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:47816/0 (socket says 172.21.15.7:47816) 2024-03-25T18:15:57.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.701+0000 7f1ee77fe640 1 -- 172.21.15.7:0/2409384389 learned_addr learned my addr 172.21.15.7:0/2409384389 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:15:57.704 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.704+0000 7f1ecffff640 10 client.?.objecter ms_handle_connect 0x7f1ee80b1b00 2024-03-25T18:15:57.704 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.704+0000 7f1ecffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.707+0000 7f1ecffff640 10 client.?.objecter ms_handle_connect 0x7f1ee80b44b0 2024-03-25T18:15:57.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.707+0000 7f1ecffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.713+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2455423278 0 0) 0x7f1ee8140c50 con 0x7f1ee80b1b00 2024-03-25T18:15:57.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.713+0000 7f1ecffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.713+0000 7f1ecffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.713+0000 7f1ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.713+0000 7f1ecffff640 10 monclient(hunting): my global_id is 4288 2024-03-25T18:15:57.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.713+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ed4004eb0 con 0x7f1ee80b1b00 2024-03-25T18:15:57.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.716+0000 7f1ecffff640 10 client.?.objecter ms_handle_connect 0x7f1ee80b4e70 2024-03-25T18:15:57.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.716+0000 7f1ecffff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:15:57.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.717+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1200070908 0 0) 0x7f1ee813cc00 con 0x7f1ee80b44b0 2024-03-25T18:15:57.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.717+0000 7f1ecffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.717+0000 7f1ecffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.717+0000 7f1ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.717+0000 7f1ecffff640 10 monclient(hunting): my global_id is 4307 2024-03-25T18:15:57.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.717+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee8140c50 con 0x7f1ee80b44b0 2024-03-25T18:15:57.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 171923183 0 0) 0x7f1ee8143600 con 0x7f1ee80b4e70 2024-03-25T18:15:57.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): _negotiate 2024-03-25T18:15:57.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:15:57.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:15:57.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): my global_id is 4299 2024-03-25T18:15:57.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ee813cc00 con 0x7f1ee80b4e70 2024-03-25T18:15:57.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 573441710 0 0) 0x7f1ed4004eb0 con 0x7f1ee80b1b00 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ee8143600 con 0x7f1ee80b1b00 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1ed8001970 con 0x7f1ee80b1b00 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): dump: 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:57.729 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.727+0000 7f1ecffff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:15:57.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 647632462 0 0) 0x7f1ee8143600 con 0x7f1ee80b1b00 2024-03-25T18:15:57.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 >> v1:172.21.15.7:6790/0 conn(0x7f1ee80b44b0 legacy=0x7f1ee81df5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 >> v1:172.21.15.7:6789/0 conn(0x7f1ee80b4e70 legacy=0x7f1ee81e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:57.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 10 monclient: _finish_hunting 0 2024-03-25T18:15:57.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 1 monclient: found mon.b 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ee8133830 con 0x7f1ee80b1b00 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 10 monclient: _finish_auth 0 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 10 monclient: _check_auth_tickets 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1ecffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 5 monclient: authenticate success, global_id 4288 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 10 monclient: _renew_subs 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee813ad40 con 0x7f1ee80b1b00 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 10 client.4288.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 10 monclient: _renew_subs 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee813b790 con 0x7f1ee80b1b00 2024-03-25T18:15:57.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.735+0000 7f1eedd04640 1 librados: init done 2024-03-25T18:15:57.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.737+0000 7f1eedd04640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.737+0000 7f1eedd04640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:57.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.737+0000 7f1eedd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:57.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.737+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee813b080 con 0x7f1ee80b1b00 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.761+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ed80047e0 con 0x7f1ee80b1b00 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.761+0000 7f1ecffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.761+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1ed8004950 con 0x7f1ee80b1b00 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.761+0000 7f1ecffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.761+0000 7f1ecffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.761+0000 7f1ecffff640 10 monclient: dump: 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.762 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.761+0000 7f1ee5ffb640 4 set_mon_vals no callback set 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.762+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f1ed801a960 con 0x7f1ee80b1b00 2024-03-25T18:15:57.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.763+0000 7f1ee6ffd640 10 monclient: get_auth_request con 0x7f1ed406de80 auth_method 0 2024-03-25T18:15:57.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.763+0000 7f1ecffff640 10 client.4288.objecter ms_handle_connect 0x7f1ed406de80 2024-03-25T18:15:57.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.765+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f1ed8054610 con 0x7f1ee80b1b00 2024-03-25T18:15:57.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.765+0000 7f1ecffff640 10 client.4288.objecter ms_dispatch 0x7f1ee8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:15:57.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.765+0000 7f1ecffff640 3 client.4288.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:15:57.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.765+0000 7f1ecffff640 3 client.4288.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:15:57.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:57.765+0000 7f1ecffff640 20 client.4288.objecter dump_active .. 0 homeless 2024-03-25T18:15:58.029 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.029+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ed804ff50 con 0x7f1ee80b1b00 2024-03-25T18:15:58.029 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.029+0000 7f1ecffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:15:58.029 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.029+0000 7f1ecffff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:15:58.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.286+0000 7f1eedd04640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:58.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.286+0000 7f1eedd04640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:58.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.286+0000 7f1eedd04640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:15:58.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.286+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ee804f730 con 0x7f1ee80b1b00 2024-03-25T18:15:58.310 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.309+0000 7f1ecffff640 1 -- 172.21.15.7:0/2409384389 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f1ed8052050 con 0x7f1ee80b1b00 2024-03-25T18:15:58.310 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.309+0000 7f1ecffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:15:58.310 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.309+0000 7f1ecffff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:15:58.310 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:15:58.310 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 10 librados: watch_flush enter 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 10 librados: watch_flush exit 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 20 client.4288.objecter shutdown clearing up homeless session... 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 10 client.4288.objecter successfully canceled tick 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 >> v1:172.21.15.7:6800/1717564767 conn(0x7f1ed406de80 legacy=0x7f1ed4070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 10 monclient: shutdown 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 >> v1:172.21.15.183:6789/0 conn(0x7f1ee80b1b00 legacy=0x7f1ee8140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.311+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 shutdown_connections 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.312+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 >> 172.21.15.7:0/2409384389 conn(0x7f1ee80617a0 msgr2=0x7f1ee812d970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.312+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 shutdown_connections 2024-03-25T18:15:58.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.312+0000 7f1eedd04640 1 -- 172.21.15.7:0/2409384389 wait complete. 2024-03-25T18:15:58.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:15:58.312+0000 7f1eedd04640 1 librados: shutdown 2024-03-25T18:15:58.324 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:05.131 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.242+0000 7fe681f15640 10 monclient: get_monmap_and_config 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.242+0000 7fe681f15640 10 monclient: build_initial_monmap 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.242+0000 7fe681f15640 10 monclient: monmap: 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:05.243890+0000 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:05.243890+0000 2024-03-25T18:16:05.243 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.243+0000 7fe681f15640 1 Processor -- start 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.243+0000 7fe681f15640 1 -- start start 2024-03-25T18:16:05.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.243+0000 7fe681f15640 10 monclient: init 2024-03-25T18:16:05.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:05.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:05.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 10 monclient(hunting): picked mon.noname-c con 0x7fe67c0b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:05.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 10 monclient(hunting): picked mon.noname-b con 0x7fe67c0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:05.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 10 monclient(hunting): picked mon.noname-a con 0x7fe67c0b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:05.245 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c0a96f0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c0ab0e0 con 0x7fe67c0b4e70 2024-03-25T18:16:05.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c145f70 con 0x7fe67c0b44b0 2024-03-25T18:16:05.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:05.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.244+0000 7fe681f15640 10 monclient(hunting): authenticate will time out at 7407.349361s 2024-03-25T18:16:05.253 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.252+0000 7fe67b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe67c0b4e70 0x7fe67c13f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:48580/0 (socket says 172.21.15.7:48580) 2024-03-25T18:16:05.253 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.252+0000 7fe67b7fe640 1 -- 172.21.15.7:0/1154961162 learned_addr learned my addr 172.21.15.7:0/1154961162 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:05.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.261+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 326114248 0 0) 0x7fe67c0ab0e0 con 0x7fe67c0b4e70 2024-03-25T18:16:05.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.261+0000 7fe67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:05.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.261+0000 7fe67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:05.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.261+0000 7fe67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:05.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.261+0000 7fe67a7fc640 10 monclient(hunting): my global_id is 4291 2024-03-25T18:16:05.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.261+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe664003520 con 0x7fe67c0b4e70 2024-03-25T18:16:05.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.264+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734448361 0 0) 0x7fe67c0a96f0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.264+0000 7fe67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:05.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.264+0000 7fe67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.264+0000 7fe67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.264+0000 7fe67a7fc640 10 monclient(hunting): my global_id is 4302 2024-03-25T18:16:05.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.264+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67c0ab0e0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.267+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891493797 0 0) 0x7fe67c145f70 con 0x7fe67c0b44b0 2024-03-25T18:16:05.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.267+0000 7fe67a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:05.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.267+0000 7fe67a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:05.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.267+0000 7fe67a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:05.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.267+0000 7fe67a7fc640 10 monclient(hunting): my global_id is 4310 2024-03-25T18:16:05.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.267+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67c0a96f0 con 0x7fe67c0b44b0 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 539686893 0 0) 0x7fe67c0ab0e0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 >> v1:172.21.15.7:6790/0 conn(0x7fe67c0b44b0 legacy=0x7fe67c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 >> v1:172.21.15.183:6789/0 conn(0x7fe67c0b4e70 legacy=0x7fe67c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 1 monclient: found mon.noname-a 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c0a9b80 con 0x7fe67c0b1b00 2024-03-25T18:16:05.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe6740019f0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe681f15640 5 monclient: authenticate success, global_id 4302 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe681f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe67a7fc640 10 monclient: dump: 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:05.272 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:05.273 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:05.273 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:05.273 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.270+0000 7fe681f15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:05.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe674001b50 con 0x7fe67c0b1b00 2024-03-25T18:16:05.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe67a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:05.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe67a7fc640 1 -- 172.21.15.7:0/1154961162 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe674001f60 con 0x7fe67c0b1b00 2024-03-25T18:16:05.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:05.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 4 set_mon_vals no callback set 2024-03-25T18:16:05.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe67a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe67a7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe67a7fc640 10 monclient: dump: 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:05.280 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:05.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 10 monclient: shutdown 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 1 -- 172.21.15.7:0/1154961162 >> v1:172.21.15.7:6789/0 conn(0x7fe67c0b1b00 legacy=0x7fe67c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 1 -- 172.21.15.7:0/1154961162 shutdown_connections 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 1 -- 172.21.15.7:0/1154961162 >> 172.21.15.7:0/1154961162 conn(0x7fe67c0617a0 msgr2=0x7fe67c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 1 -- 172.21.15.7:0/1154961162 shutdown_connections 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.279+0000 7fe681f15640 1 -- 172.21.15.7:0/1154961162 wait complete. 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.280+0000 7fe681f15640 10 monclient: build_initial_monmap 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.280+0000 7fe681f15640 10 monclient: monmap: 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:05.282 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.280+0000 7fe681f15640 1 librados: starting msgr at 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.280+0000 7fe681f15640 1 librados: starting objecter 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.280+0000 7fe681f15640 1 Processor -- start 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 1 -- start start 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 1 librados: setting wanted keys 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 1 librados: calling monclient init 2024-03-25T18:16:05.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient: init 2024-03-25T18:16:05.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:05.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:16:05.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient(hunting): picked mon.noname-c con 0x7fe67c0b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:05.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient(hunting): picked mon.noname-a con 0x7fe67c0b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:05.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient(hunting): picked mon.noname-b con 0x7fe67c0b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:05.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c13b6d0 con 0x7fe67c0b44b0 2024-03-25T18:16:05.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c12dc60 con 0x7fe67c0b4e70 2024-03-25T18:16:05.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe67c13b2d0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:05.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.281+0000 7fe681f15640 10 monclient(hunting): authenticate will time out at 7407.386514s 2024-03-25T18:16:05.290 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.290+0000 7fe67b7fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fe67c0b1b00 0x7fe67c13abc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:38694/0 (socket says 172.21.15.7:38694) 2024-03-25T18:16:05.291 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.290+0000 7fe67b7fe640 1 -- 172.21.15.7:0/2486213463 learned_addr learned my addr 172.21.15.7:0/2486213463 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:05.291 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.291+0000 7fe663fff640 10 client.?.objecter ms_handle_connect 0x7fe67c0b1b00 2024-03-25T18:16:05.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.291+0000 7fe663fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.296+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4044782602 0 0) 0x7fe67c13b2d0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.296+0000 7fe663fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.296+0000 7fe663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.296+0000 7fe663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.296+0000 7fe663fff640 10 monclient(hunting): my global_id is 4313 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.296+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe668004eb0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.297+0000 7fe663fff640 10 client.?.objecter ms_handle_connect 0x7fe67c0b4e70 2024-03-25T18:16:05.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.297+0000 7fe663fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:05.305 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.304+0000 7fe663fff640 10 client.?.objecter ms_handle_connect 0x7fe67c0b44b0 2024-03-25T18:16:05.305 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.304+0000 7fe663fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:05.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.309+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3905772191 0 0) 0x7fe67c12dc60 con 0x7fe67c0b4e70 2024-03-25T18:16:05.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.309+0000 7fe663fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:05.310 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.309+0000 7fe663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:05.310 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.309+0000 7fe663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:05.310 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.309+0000 7fe663fff640 10 monclient(hunting): my global_id is 4294 2024-03-25T18:16:05.310 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.309+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67c13b2d0 con 0x7fe67c0b4e70 2024-03-25T18:16:05.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.313+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 457709354 0 0) 0x7fe67c13b6d0 con 0x7fe67c0b44b0 2024-03-25T18:16:05.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.313+0000 7fe663fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:05.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.313+0000 7fe663fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:05.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.313+0000 7fe663fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:05.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.313+0000 7fe663fff640 10 monclient(hunting): my global_id is 4305 2024-03-25T18:16:05.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.313+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe67c12dc60 con 0x7fe67c0b44b0 2024-03-25T18:16:05.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.319+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4083995413 0 0) 0x7fe668004eb0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.320+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe67c13b6d0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.320+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe66c001830 con 0x7fe67c0b1b00 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.320+0000 7fe663fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.320+0000 7fe663fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.320+0000 7fe663fff640 10 monclient(hunting): dump: 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:05.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.320+0000 7fe663fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:05.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3742287929 0 0) 0x7fe67c13b6d0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 >> v1:172.21.15.183:6789/0 conn(0x7fe67c0b4e70 legacy=0x7fe67c1daec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 >> v1:172.21.15.7:6789/0 conn(0x7fe67c0b44b0 legacy=0x7fe67c12cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 1 monclient: found mon.c 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe67c13b460 con 0x7fe67c0b1b00 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe663fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 5 monclient: authenticate success, global_id 4313 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 10 monclient: _renew_subs 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe67c13cc80 con 0x7fe67c0b1b00 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 10 monclient: _renew_subs 2024-03-25T18:16:05.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:05.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe67c12d800 con 0x7fe67c0b1b00 2024-03-25T18:16:05.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.343+0000 7fe681f15640 1 librados: init done 2024-03-25T18:16:05.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.344+0000 7fe681f15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:05.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.344+0000 7fe681f15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:05.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.344+0000 7fe681f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:05.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.344+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe67c13a000 con 0x7fe67c0b1b00 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.688+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe66c001ce0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.688+0000 7fe663fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.688+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe66c004980 con 0x7fe67c0b1b00 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.688+0000 7fe663fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.688+0000 7fe663fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.688+0000 7fe663fff640 10 monclient: dump: 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:05.690 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:05.691 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.688+0000 7fe679ffb640 4 set_mon_vals no callback set 2024-03-25T18:16:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.689+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fe66c004ba0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.690+0000 7fe67affd640 10 monclient: get_auth_request con 0x7fe66806de30 auth_method 0 2024-03-25T18:16:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.690+0000 7fe663fff640 10 client.4313.objecter ms_handle_connect 0x7fe66806de30 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fe66c0547c0 con 0x7fe67c0b1b00 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 10 client.4313.objecter ms_dispatch 0x7fe67c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe66c050100 con 0x7fe67c0b1b00 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.696+0000 7fe663fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:05.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.904+0000 7fe681f15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.904+0000 7fe681f15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.904+0000 7fe681f15640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:05.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.904+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe67c12da50 con 0x7fe67c0b1b00 2024-03-25T18:16:05.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.942+0000 7fe663fff640 1 -- 172.21.15.7:0/2486213463 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fe66c052200 con 0x7fe67c0b1b00 2024-03-25T18:16:05.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.942+0000 7fe663fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:05.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.942+0000 7fe663fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:05.943 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:05.943 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:05.946 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 10 librados: watch_flush enter 2024-03-25T18:16:05.946 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 10 librados: watch_flush exit 2024-03-25T18:16:05.946 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-25T18:16:05.946 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 10 client.4313.objecter successfully canceled tick 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 >> v1:172.21.15.7:6800/1717564767 conn(0x7fe66806de30 legacy=0x7fe6680702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 10 monclient: shutdown 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 >> v1:172.21.15.7:6790/0 conn(0x7fe67c0b1b00 legacy=0x7fe67c13abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 shutdown_connections 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 >> 172.21.15.7:0/2486213463 conn(0x7fe67c0617a0 msgr2=0x7fe67c0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 shutdown_connections 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 1 -- 172.21.15.7:0/2486213463 wait complete. 2024-03-25T18:16:05.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:05.945+0000 7fe681f15640 1 librados: shutdown 2024-03-25T18:16:05.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:08.238 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:14.875 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.245 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:14.882 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.282 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:14.919 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.304 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:14.931 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.308 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:14.935 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.312 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:14.950 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.322 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:14.959 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.336 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:14.963 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:08.350 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:14.988 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:12.766 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.882+0000 7f05084c8640 10 monclient: get_monmap_and_config 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.882+0000 7f05084c8640 10 monclient: build_initial_monmap 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.882+0000 7f05084c8640 10 monclient: monmap: 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:12.883488+0000 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:12.883488+0000 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:12.883 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:12.884 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:12.884 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:12.884 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:12.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.883+0000 7f05084c8640 1 Processor -- start 2024-03-25T18:16:12.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.883+0000 7f05084c8640 1 -- start start 2024-03-25T18:16:12.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.883+0000 7f05084c8640 10 monclient: init 2024-03-25T18:16:12.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:12.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:12.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 10 monclient(hunting): picked mon.noname-b con 0x7f05000b1ed0 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:12.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 10 monclient(hunting): picked mon.noname-c con 0x7f0500071db0 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:12.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 10 monclient(hunting): picked mon.noname-a con 0x7f0500143f90 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:12.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050006eef0 con 0x7f0500143f90 2024-03-25T18:16:12.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050006f1d0 con 0x7f0500071db0 2024-03-25T18:16:12.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050006dfe0 con 0x7f05000b1ed0 2024-03-25T18:16:12.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.884+0000 7f05084c8640 10 monclient(hunting): authenticate will time out at 7414.989275s 2024-03-25T18:16:12.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.896+0000 7f0506264640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f05000b1ed0 0x7f0500143880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:40340/0 (socket says 172.21.15.7:40340) 2024-03-25T18:16:12.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.897+0000 7f0506264640 1 -- 172.21.15.7:0/1732523274 learned_addr learned my addr 172.21.15.7:0/1732523274 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:12.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.918+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1292245496 0 0) 0x7f050006dfe0 con 0x7f05000b1ed0 2024-03-25T18:16:12.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.918+0000 7f0505262640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.918+0000 7f0505262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.918+0000 7f0505262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.918+0000 7f0505262640 10 monclient(hunting): my global_id is 4316 2024-03-25T18:16:12.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.918+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04ec003520 con 0x7f05000b1ed0 2024-03-25T18:16:12.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.925+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705275872 0 0) 0x7f050006f1d0 con 0x7f0500071db0 2024-03-25T18:16:12.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.925+0000 7f0505262640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.925+0000 7f0505262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.925+0000 7f0505262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.925+0000 7f0505262640 10 monclient(hunting): my global_id is 4297 2024-03-25T18:16:12.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.925+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050006dfe0 con 0x7f0500071db0 2024-03-25T18:16:12.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.932+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2322210189 0 0) 0x7f050006eef0 con 0x7f0500143f90 2024-03-25T18:16:12.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.932+0000 7f0505262640 10 monclient(hunting): _negotiate 2024-03-25T18:16:12.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.932+0000 7f0505262640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:12.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.932+0000 7f0505262640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:12.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.932+0000 7f0505262640 10 monclient(hunting): my global_id is 4308 2024-03-25T18:16:12.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.932+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050006f1d0 con 0x7f0500143f90 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.951+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1714411635 0 0) 0x7f050006dfe0 con 0x7f0500071db0 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 >> v1:172.21.15.7:6790/0 conn(0x7f05000b1ed0 legacy=0x7f0500143880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 >> v1:172.21.15.7:6789/0 conn(0x7f0500143f90 legacy=0x7f0500152000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 10 monclient: _finish_hunting 0 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 1 monclient: found mon.noname-c 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f050006e170 con 0x7f0500071db0 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 10 monclient: _finish_auth 0 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 10 monclient: _check_auth_tickets 2024-03-25T18:16:12.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04f0001670 con 0x7f0500071db0 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f0505262640 10 monclient: dump: 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:12.954 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:12.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f05084c8640 5 monclient: authenticate success, global_id 4297 2024-03-25T18:16:12.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.952+0000 7f05084c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f00017d0 con 0x7f0500071db0 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f0505262640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f0505262640 1 -- 172.21.15.7:0/1732523274 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04f0001b60 con 0x7f0500071db0 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f05084c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f05084c8640 4 set_mon_vals no callback set 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f0505262640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f0505262640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.972+0000 7f0505262640 10 monclient: dump: 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:12.973 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:12.974 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:12.974 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:12.974 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:12.974 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:12.974 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:12.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.973+0000 7f05084c8640 10 monclient: shutdown 2024-03-25T18:16:12.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.973+0000 7f05084c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:12.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.973+0000 7f05084c8640 1 -- 172.21.15.7:0/1732523274 >> v1:172.21.15.183:6789/0 conn(0x7f0500071db0 legacy=0x7f0500146740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:12.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.973+0000 7f05084c8640 1 -- 172.21.15.7:0/1732523274 shutdown_connections 2024-03-25T18:16:12.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.973+0000 7f05084c8640 1 -- 172.21.15.7:0/1732523274 >> 172.21.15.7:0/1732523274 conn(0x7f05000617a0 msgr2=0x7f0500061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:12.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.973+0000 7f05084c8640 1 -- 172.21.15.7:0/1732523274 shutdown_connections 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.973+0000 7f05084c8640 1 -- 172.21.15.7:0/1732523274 wait complete. 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.974+0000 7f05084c8640 10 monclient: build_initial_monmap 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.974+0000 7f05084c8640 10 monclient: monmap: 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.974+0000 7f05084c8640 1 librados: starting msgr at 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.974+0000 7f05084c8640 1 librados: starting objecter 2024-03-25T18:16:12.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 1 Processor -- start 2024-03-25T18:16:12.977 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 1 -- start start 2024-03-25T18:16:12.977 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 1 librados: setting wanted keys 2024-03-25T18:16:12.977 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 1 librados: calling monclient init 2024-03-25T18:16:12.977 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 10 monclient: init 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 10 monclient(hunting): picked mon.noname-c con 0x7f0500071db0 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.975+0000 7f05084c8640 10 monclient(hunting): picked mon.noname-b con 0x7f05000b1ed0 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.976+0000 7f05084c8640 10 monclient(hunting): picked mon.noname-a con 0x7f0500143f90 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.976+0000 7f05084c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050014a420 con 0x7f0500143f90 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.976+0000 7f05084c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f050014a100 con 0x7f05000b1ed0 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.976+0000 7f05084c8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05000b6870 con 0x7f0500071db0 2024-03-25T18:16:12.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.976+0000 7f05084c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:12.979 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.976+0000 7f05084c8640 10 monclient(hunting): authenticate will time out at 7415.080826s 2024-03-25T18:16:12.992 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.991+0000 7f0506a65640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0500143f90 0x7f05001debb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36698/0 (socket says 172.21.15.7:36698) 2024-03-25T18:16:12.992 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.991+0000 7f0506a65640 1 -- 172.21.15.7:0/1896522746 learned_addr learned my addr 172.21.15.7:0/1896522746 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:12.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.992+0000 7f04f6ffd640 10 client.?.objecter ms_handle_connect 0x7f0500143f90 2024-03-25T18:16:12.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.992+0000 7f04f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:12.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.996+0000 7f04f6ffd640 10 client.?.objecter ms_handle_connect 0x7f05000b1ed0 2024-03-25T18:16:12.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:12.996+0000 7f04f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:13.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.002+0000 7f04f6ffd640 10 client.?.objecter ms_handle_connect 0x7f0500071db0 2024-03-25T18:16:13.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.002+0000 7f04f6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883583831 0 0) 0x7f050014a100 con 0x7f05000b1ed0 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): my global_id is 4300 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e4004eb0 con 0x7f05000b1ed0 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4186735302 0 0) 0x7f050014a420 con 0x7f0500143f90 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:13.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:13.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 10 monclient(hunting): my global_id is 4311 2024-03-25T18:16:13.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.010+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050014a100 con 0x7f0500143f90 2024-03-25T18:16:13.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.018+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 157311238 0 0) 0x7f05000b6870 con 0x7f0500071db0 2024-03-25T18:16:13.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.018+0000 7f04f6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:13.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.018+0000 7f04f6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:13.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.018+0000 7f04f6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:13.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.018+0000 7f04f6ffd640 10 monclient(hunting): my global_id is 4319 2024-03-25T18:16:13.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.018+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f050014a420 con 0x7f0500071db0 2024-03-25T18:16:13.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.027+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1166182224 0 0) 0x7f04e4004eb0 con 0x7f05000b1ed0 2024-03-25T18:16:13.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.027+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f05000b6870 con 0x7f05000b1ed0 2024-03-25T18:16:13.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.027+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04f0001ae0 con 0x7f05000b1ed0 2024-03-25T18:16:13.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.027+0000 7f04f6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:13.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.027+0000 7f04f6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.027+0000 7f04f6ffd640 10 monclient(hunting): dump: 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:13.029 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.027+0000 7f04f6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3922321781 0 0) 0x7f05000b6870 con 0x7f05000b1ed0 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 >> v1:172.21.15.7:6790/0 conn(0x7f0500071db0 legacy=0x7f05001499f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 >> v1:172.21.15.7:6789/0 conn(0x7f0500143f90 legacy=0x7f05001debb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 1 monclient: found mon.b 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0500071100 con 0x7f05000b1ed0 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f04f6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 5 monclient: authenticate success, global_id 4300 2024-03-25T18:16:13.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 10 monclient: _renew_subs 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0500071350 con 0x7f05000b1ed0 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 10 client.4300.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 10 monclient: _renew_subs 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f050013ab90 con 0x7f05000b1ed0 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.034+0000 7f05084c8640 1 librados: init done 2024-03-25T18:16:13.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.036+0000 7f05084c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:13.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.036+0000 7f05084c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:13.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.036+0000 7f05084c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:13.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.036+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05000b6690 con 0x7f05000b1ed0 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.312+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04f00038a0 con 0x7f05000b1ed0 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.312+0000 7f04f6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.312+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04f0003bb0 con 0x7f05000b1ed0 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.312+0000 7f04f6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.312+0000 7f04f6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.312+0000 7f04f6ffd640 10 monclient: dump: 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:13.313 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:13.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.312+0000 7f0504a61640 4 set_mon_vals no callback set 2024-03-25T18:16:13.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.313+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f04f0019b80 con 0x7f05000b1ed0 2024-03-25T18:16:13.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.314+0000 7f0506264640 10 monclient: get_auth_request con 0x7f04e406dda0 auth_method 0 2024-03-25T18:16:13.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.314+0000 7f04f6ffd640 10 client.4300.objecter ms_handle_connect 0x7f04e406dda0 2024-03-25T18:16:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f04f0053780 con 0x7f05000b1ed0 2024-03-25T18:16:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 10 client.4300.objecter ms_dispatch 0x7f0500000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 3 client.4300.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 3 client.4300.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 20 client.4300.objecter dump_active .. 0 homeless 2024-03-25T18:16:13.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04f004f0c0 con 0x7f05000b1ed0 2024-03-25T18:16:13.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:13.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.333+0000 7f04f6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:13.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.561+0000 7f05084c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:13.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.561+0000 7f05084c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:13.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.561+0000 7f05084c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:13.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.561+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f050006dbd0 con 0x7f05000b1ed0 2024-03-25T18:16:13.590 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.589+0000 7f04f6ffd640 1 -- 172.21.15.7:0/1896522746 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f04f00511c0 con 0x7f05000b1ed0 2024-03-25T18:16:13.590 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.590+0000 7f04f6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:13.590 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.590+0000 7f04f6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:13.591 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:13.591 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:13.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 10 librados: watch_flush enter 2024-03-25T18:16:13.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 10 librados: watch_flush exit 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 20 client.4300.objecter shutdown clearing up homeless session... 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 10 client.4300.objecter successfully canceled tick 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 >> v1:172.21.15.7:6800/1717564767 conn(0x7f04e406dda0 legacy=0x7f04e4070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 10 monclient: shutdown 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 >> v1:172.21.15.183:6789/0 conn(0x7f05000b1ed0 legacy=0x7f050013a2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 shutdown_connections 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 >> 172.21.15.7:0/1896522746 conn(0x7f05000617a0 msgr2=0x7f0500141410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 shutdown_connections 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 1 -- 172.21.15.7:0/1896522746 wait complete. 2024-03-25T18:16:13.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:13.592+0000 7f05084c8640 1 librados: shutdown 2024-03-25T18:16:13.605 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:20.411 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:20.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.514+0000 7fe0db11c640 10 monclient: get_monmap_and_config 2024-03-25T18:16:20.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.514+0000 7fe0db11c640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.514+0000 7fe0db11c640 10 monclient: monmap: 2024-03-25T18:16:20.515 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:20.515 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:20.515 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:20.515689+0000 2024-03-25T18:16:20.516 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:20.515689+0000 2024-03-25T18:16:20.516 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.516 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:20.516 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:20.516 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:20.516 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:20.516 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:20.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.515+0000 7fe0db11c640 1 Processor -- start 2024-03-25T18:16:20.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.515+0000 7fe0db11c640 1 -- start start 2024-03-25T18:16:20.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.515+0000 7fe0db11c640 10 monclient: init 2024-03-25T18:16:20.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d40b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d40b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0d40b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40a96f0 con 0x7fe0d40b1b00 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d40ab0e0 con 0x7fe0d40b4e70 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d4145f70 con 0x7fe0d40b44b0 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.516+0000 7fe0db11c640 10 monclient(hunting): authenticate will time out at 7422.621166s 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.517+0000 7fe0d96b9640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe0d40b1b00 0x7fe0d414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41390/0 (socket says 172.21.15.7:41390) 2024-03-25T18:16:20.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.517+0000 7fe0d96b9640 1 -- 172.21.15.7:0/2428625407 learned_addr learned my addr 172.21.15.7:0/2428625407 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:20.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.534+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3105516523 0 0) 0x7fe0d4145f70 con 0x7fe0d40b44b0 2024-03-25T18:16:20.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.534+0000 7fe0d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.534+0000 7fe0d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.535 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.534+0000 7fe0d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.535 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.534+0000 7fe0d37fe640 10 monclient(hunting): my global_id is 4322 2024-03-25T18:16:20.535 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.534+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0c4003520 con 0x7fe0d40b44b0 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2173776745 0 0) 0x7fe0c4003520 con 0x7fe0d40b44b0 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 >> v1:172.21.15.183:6789/0 conn(0x7fe0d40b4e70 legacy=0x7fe0d413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 >> v1:172.21.15.7:6789/0 conn(0x7fe0d40b1b00 legacy=0x7fe0d414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 1 monclient: found mon.noname-b 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d40a9b80 con 0x7fe0d40b44b0 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe0c0001690 con 0x7fe0d40b44b0 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0d37fe640 10 monclient: dump: 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:20.556 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:20.557 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0db11c640 5 monclient: authenticate success, global_id 4322 2024-03-25T18:16:20.557 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.554+0000 7fe0db11c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0c00017f0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0d37fe640 1 -- 172.21.15.7:0/2428625407 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe0c0001c00 con 0x7fe0d40b44b0 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0db11c640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0db11c640 4 set_mon_vals no callback set 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0d37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.578+0000 7fe0d37fe640 10 monclient: dump: 2024-03-25T18:16:20.579 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:20.580 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:20.581 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.579+0000 7fe0db11c640 10 monclient: shutdown 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.579+0000 7fe0db11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.579+0000 7fe0db11c640 1 -- 172.21.15.7:0/2428625407 >> v1:172.21.15.7:6790/0 conn(0x7fe0d40b44b0 legacy=0x7fe0d4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.579+0000 7fe0db11c640 1 -- 172.21.15.7:0/2428625407 shutdown_connections 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.579+0000 7fe0db11c640 1 -- 172.21.15.7:0/2428625407 >> 172.21.15.7:0/2428625407 conn(0x7fe0d40617a0 msgr2=0x7fe0d4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.579+0000 7fe0db11c640 1 -- 172.21.15.7:0/2428625407 shutdown_connections 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.579+0000 7fe0db11c640 1 -- 172.21.15.7:0/2428625407 wait complete. 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 10 monclient: build_initial_monmap 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 10 monclient: monmap: 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:20.583 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 1 librados: starting msgr at 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 1 librados: starting objecter 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 1 Processor -- start 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 1 -- start start 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 1 librados: setting wanted keys 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 1 librados: calling monclient init 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.580+0000 7fe0db11c640 10 monclient: init 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-c con 0x7fe0d40b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:20.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-b con 0x7fe0d40b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:20.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 10 monclient(hunting): picked mon.noname-a con 0x7fe0d40b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:20.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d4136d70 con 0x7fe0d40b4e70 2024-03-25T18:16:20.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d41430e0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d4136fe0 con 0x7fe0d40b1b00 2024-03-25T18:16:20.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:20.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.581+0000 7fe0db11c640 10 monclient(hunting): authenticate will time out at 7422.686584s 2024-03-25T18:16:20.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.599+0000 7fe0d3fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe0d40b44b0 0x7fe0d41417a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:51342/0 (socket says 172.21.15.7:51342) 2024-03-25T18:16:20.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.599+0000 7fe0d3fff640 1 -- 172.21.15.7:0/3893814941 learned_addr learned my addr 172.21.15.7:0/3893814941 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:20.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.601+0000 7fe0d17fa640 10 client.?.objecter ms_handle_connect 0x7fe0d40b44b0 2024-03-25T18:16:20.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.601+0000 7fe0d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.604+0000 7fe0d17fa640 10 client.?.objecter ms_handle_connect 0x7fe0d40b1b00 2024-03-25T18:16:20.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.604+0000 7fe0d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.612 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.611+0000 7fe0d17fa640 10 client.?.objecter ms_handle_connect 0x7fe0d40b4e70 2024-03-25T18:16:20.612 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.611+0000 7fe0d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:20.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.617+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779212738 0 0) 0x7fe0d41430e0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.617+0000 7fe0d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.617+0000 7fe0d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.617+0000 7fe0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.617+0000 7fe0d17fa640 10 monclient(hunting): my global_id is 4306 2024-03-25T18:16:20.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.617+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b8004eb0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.619+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427129594 0 0) 0x7fe0d4136fe0 con 0x7fe0d40b1b00 2024-03-25T18:16:20.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.619+0000 7fe0d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.619+0000 7fe0d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.619+0000 7fe0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.619+0000 7fe0d17fa640 10 monclient(hunting): my global_id is 4325 2024-03-25T18:16:20.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.619+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d41430e0 con 0x7fe0d40b1b00 2024-03-25T18:16:20.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.625+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1448068296 0 0) 0x7fe0b8004eb0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.625+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d4136fe0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.626+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 152525005 0 0) 0x7fe0d4136d70 con 0x7fe0d40b4e70 2024-03-25T18:16:20.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.626+0000 7fe0d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:16:20.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.626+0000 7fe0d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:20.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.626+0000 7fe0d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:20.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.626+0000 7fe0d17fa640 10 monclient(hunting): my global_id is 4314 2024-03-25T18:16:20.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.626+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b8004eb0 con 0x7fe0d40b4e70 2024-03-25T18:16:20.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.641+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe0c00035f0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.641+0000 7fe0d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.641+0000 7fe0d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.641+0000 7fe0d17fa640 10 monclient(hunting): dump: 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:20.642 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:20.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.641+0000 7fe0d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:20.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.645+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4068587436 0 0) 0x7fe0d41430e0 con 0x7fe0d40b1b00 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.645+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0d4136d70 con 0x7fe0d40b1b00 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.645+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe0c8001800 con 0x7fe0d40b1b00 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.645+0000 7fe0d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.645+0000 7fe0d17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.645+0000 7fe0d17fa640 10 monclient(hunting): dump: 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:20.646 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:20.647 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:20.647 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1520877180 0 0) 0x7fe0d4136fe0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 >> v1:172.21.15.7:6790/0 conn(0x7fe0d40b1b00 legacy=0x7fe0d41376e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 >> v1:172.21.15.7:6789/0 conn(0x7fe0d40b4e70 legacy=0x7fe0d41e25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 1 monclient: found mon.b 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d41372c0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 10 monclient: _finish_auth 0 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.649+0000 7fe0d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 5 monclient: authenticate success, global_id 4306 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 10 monclient: _renew_subs 2024-03-25T18:16:20.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:20.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d4136080 con 0x7fe0d40b44b0 2024-03-25T18:16:20.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 10 client.4306.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:20.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 10 monclient: _renew_subs 2024-03-25T18:16:20.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:20.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d41353a0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.650+0000 7fe0db11c640 1 librados: init done 2024-03-25T18:16:20.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.651+0000 7fe0db11c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.651+0000 7fe0db11c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.651+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:20.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.651+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d404f730 con 0x7fe0d40b44b0 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.929+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0c0003ac0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.929+0000 7fe0d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.929+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe0c0003e90 con 0x7fe0d40b44b0 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.929+0000 7fe0d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.929+0000 7fe0d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.929+0000 7fe0d17fa640 10 monclient: dump: 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:20.930 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:20.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.929+0000 7fe0d2ffd640 4 set_mon_vals no callback set 2024-03-25T18:16:20.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.930+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fe0c00040b0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.930+0000 7fe0d8eb8640 10 monclient: get_auth_request con 0x7fe0b806df30 auth_method 0 2024-03-25T18:16:20.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.931+0000 7fe0d17fa640 10 client.4306.objecter ms_handle_connect 0x7fe0b806df30 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fe0c0054990 con 0x7fe0d40b44b0 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 10 client.4306.objecter ms_dispatch 0x7fe0d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 3 client.4306.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 3 client.4306.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 20 client.4306.objecter dump_active .. 0 homeless 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe0c00504a0 con 0x7fe0d40b44b0 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:20.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:20.935+0000 7fe0d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:21.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.188+0000 7fe0db11c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:21.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.188+0000 7fe0db11c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:21.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.188+0000 7fe0db11c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:21.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.188+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0d41358d0 con 0x7fe0d40b44b0 2024-03-25T18:16:21.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.216+0000 7fe0d17fa640 1 -- 172.21.15.7:0/3893814941 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fe0d41358d0 con 0x7fe0d40b44b0 2024-03-25T18:16:21.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.216+0000 7fe0d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:21.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.216+0000 7fe0d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:21.217 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:21.217 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:21.220 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 10 librados: watch_flush enter 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 10 librados: watch_flush exit 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 20 client.4306.objecter shutdown clearing up homeless session... 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 10 client.4306.objecter successfully canceled tick 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 >> v1:172.21.15.7:6800/1717564767 conn(0x7fe0b806df30 legacy=0x7fe0b80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 10 monclient: shutdown 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 >> v1:172.21.15.183:6789/0 conn(0x7fe0d40b44b0 legacy=0x7fe0d41417a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 shutdown_connections 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 >> 172.21.15.7:0/3893814941 conn(0x7fe0d40617a0 msgr2=0x7fe0d40b5d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 shutdown_connections 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 1 -- 172.21.15.7:0/3893814941 wait complete. 2024-03-25T18:16:21.221 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:21.219+0000 7fe0db11c640 1 librados: shutdown 2024-03-25T18:16:21.232 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:28.039 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:28.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.156+0000 7f97c5dd6640 10 monclient: get_monmap_and_config 2024-03-25T18:16:28.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.156+0000 7f97c5dd6640 10 monclient: build_initial_monmap 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.156+0000 7f97c5dd6640 10 monclient: monmap: 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:28.157960+0000 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:28.157960+0000 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:28.158 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:28.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.157+0000 7f97c5dd6640 1 Processor -- start 2024-03-25T18:16:28.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.158+0000 7f97c5dd6640 1 -- start start 2024-03-25T18:16:28.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.158+0000 7f97c5dd6640 10 monclient: init 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.158+0000 7f97c5dd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.158+0000 7f97c5dd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.158+0000 7f97c5dd6640 10 monclient(hunting): picked mon.noname-b con 0x7f97c00b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.158+0000 7f97c5dd6640 10 monclient(hunting): picked mon.noname-c con 0x7f97c00b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.159+0000 7f97c5dd6640 10 monclient(hunting): picked mon.noname-a con 0x7f97c00b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.159+0000 7f97c5dd6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c00a96f0 con 0x7f97c00b1b00 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.159+0000 7f97c5dd6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c00ab0e0 con 0x7f97c00b44b0 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.159+0000 7f97c5dd6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c0145f70 con 0x7f97c00b4e70 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.159+0000 7f97c5dd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:28.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.159+0000 7f97c5dd6640 10 monclient(hunting): authenticate will time out at 7430.263827s 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.176+0000 7f97beffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f97c00b44b0 0x7f97c0145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:36900/0 (socket says 172.21.15.7:36900) 2024-03-25T18:16:28.177 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.176+0000 7f97beffd640 1 -- 172.21.15.7:0/1431314747 learned_addr learned my addr 172.21.15.7:0/1431314747 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:28.188 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.188+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2648165939 0 0) 0x7f97c00ab0e0 con 0x7f97c00b44b0 2024-03-25T18:16:28.188 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.188+0000 7f97be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:28.188 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.188+0000 7f97be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:28.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.188+0000 7f97be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:28.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.188+0000 7f97be7fc640 10 monclient(hunting): my global_id is 4309 2024-03-25T18:16:28.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.188+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97a8003520 con 0x7f97c00b44b0 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3961971998 0 0) 0x7f97a8003520 con 0x7f97c00b44b0 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 >> v1:172.21.15.7:6790/0 conn(0x7f97c00b4e70 legacy=0x7f97c013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 >> v1:172.21.15.7:6789/0 conn(0x7f97c00b1b00 legacy=0x7f97c014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 1 monclient: found mon.noname-c 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:16:28.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c00a9b80 con 0x7f97c00b44b0 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f97b4001690 con 0x7f97c00b44b0 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97be7fc640 10 monclient: dump: 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.198 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:28.199 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:28.199 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:28.199 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:28.199 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:28.199 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:28.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97c5dd6640 5 monclient: authenticate success, global_id 4309 2024-03-25T18:16:28.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.196+0000 7f97c5dd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97b4001860 con 0x7f97c00b44b0 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97be7fc640 1 -- 172.21.15.7:0/1431314747 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f97b4001cb0 con 0x7f97c00b44b0 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97c5dd6640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97c5dd6640 4 set_mon_vals no callback set 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.201+0000 7f97be7fc640 10 monclient: dump: 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:28.202 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:28.203 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:28.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.202+0000 7f97c5dd6640 10 monclient: shutdown 2024-03-25T18:16:28.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.202+0000 7f97c5dd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:28.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.202+0000 7f97c5dd6640 1 -- 172.21.15.7:0/1431314747 >> v1:172.21.15.183:6789/0 conn(0x7f97c00b44b0 legacy=0x7f97c0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.202+0000 7f97c5dd6640 1 -- 172.21.15.7:0/1431314747 shutdown_connections 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.202+0000 7f97c5dd6640 1 -- 172.21.15.7:0/1431314747 >> 172.21.15.7:0/1431314747 conn(0x7f97c00617a0 msgr2=0x7f97c0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.202+0000 7f97c5dd6640 1 -- 172.21.15.7:0/1431314747 shutdown_connections 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.202+0000 7f97c5dd6640 1 -- 172.21.15.7:0/1431314747 wait complete. 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.203+0000 7f97c5dd6640 10 monclient: build_initial_monmap 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.203+0000 7f97c5dd6640 10 monclient: monmap: 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:28.205 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.203+0000 7f97c5dd6640 1 librados: starting msgr at 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.203+0000 7f97c5dd6640 1 librados: starting objecter 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.203+0000 7f97c5dd6640 1 Processor -- start 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 1 -- start start 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 1 librados: setting wanted keys 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 1 librados: calling monclient init 2024-03-25T18:16:28.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient: init 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient(hunting): picked mon.noname-b con 0x7f97c00b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient(hunting): picked mon.noname-a con 0x7f97c00b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient(hunting): picked mon.noname-c con 0x7f97c00b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c01379b0 con 0x7f97c00b44b0 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c0137c90 con 0x7f97c00b1b00 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97c0136b80 con 0x7f97c00b4e70 2024-03-25T18:16:28.207 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:28.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.204+0000 7f97c5dd6640 10 monclient(hunting): authenticate will time out at 7430.309515s 2024-03-25T18:16:28.215 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.214+0000 7f97beffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f97c00b44b0 0x7f97c0133ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:34226/0 (socket says 172.21.15.7:34226) 2024-03-25T18:16:28.215 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.214+0000 7f97beffd640 1 -- 172.21.15.7:0/4275560396 learned_addr learned my addr 172.21.15.7:0/4275560396 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:28.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.216+0000 7f97a7fff640 10 client.?.objecter ms_handle_connect 0x7f97c00b44b0 2024-03-25T18:16:28.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.216+0000 7f97a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:28.218 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.218+0000 7f97a7fff640 10 client.?.objecter ms_handle_connect 0x7f97c00b4e70 2024-03-25T18:16:28.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.218+0000 7f97a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:28.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.218+0000 7f97a7fff640 10 client.?.objecter ms_handle_connect 0x7f97c00b1b00 2024-03-25T18:16:28.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.218+0000 7f97a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:28.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.237+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2362410237 0 0) 0x7f97c01379b0 con 0x7f97c00b44b0 2024-03-25T18:16:28.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.237+0000 7f97a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:28.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.237+0000 7f97a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:28.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.237+0000 7f97a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:28.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.237+0000 7f97a7fff640 10 monclient(hunting): my global_id is 4317 2024-03-25T18:16:28.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.237+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97ac004eb0 con 0x7f97c00b44b0 2024-03-25T18:16:28.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.240+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294814193 0 0) 0x7f97c0137c90 con 0x7f97c00b1b00 2024-03-25T18:16:28.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.240+0000 7f97a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:28.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.240+0000 7f97a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:28.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.240+0000 7f97a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:28.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.240+0000 7f97a7fff640 10 monclient(hunting): my global_id is 4312 2024-03-25T18:16:28.241 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.240+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c01379b0 con 0x7f97c00b1b00 2024-03-25T18:16:28.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.243+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1738558785 0 0) 0x7f97c0136b80 con 0x7f97c00b4e70 2024-03-25T18:16:28.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.243+0000 7f97a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:28.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.243+0000 7f97a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:28.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.243+0000 7f97a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:28.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.243+0000 7f97a7fff640 10 monclient(hunting): my global_id is 4331 2024-03-25T18:16:28.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.243+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97c0137c90 con 0x7f97c00b4e70 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.246+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 936279535 0 0) 0x7f97c01379b0 con 0x7f97c00b1b00 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.246+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97c0136b80 con 0x7f97c00b1b00 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.246+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f97b0001810 con 0x7f97c00b1b00 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.246+0000 7f97a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.246+0000 7f97a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.246+0000 7f97a7fff640 10 monclient(hunting): dump: 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:28.247 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:28.248 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:28.248 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:28.248 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:28.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.246+0000 7f97a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:28.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4095571722 0 0) 0x7f97c0136b80 con 0x7f97c00b1b00 2024-03-25T18:16:28.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 >> v1:172.21.15.7:6790/0 conn(0x7f97c00b4e70 legacy=0x7f97c01380b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 >> v1:172.21.15.7:6789/0 conn(0x7f97c00b44b0 legacy=0x7f97c0133ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 1 monclient: found mon.b 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c0139460 con 0x7f97c00b1b00 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 5 monclient: authenticate success, global_id 4312 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 10 monclient: _renew_subs 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c0139730 con 0x7f97c00b1b00 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 10 monclient: _renew_subs 2024-03-25T18:16:28.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:28.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c01347e0 con 0x7f97c00b1b00 2024-03-25T18:16:28.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.254+0000 7f97c5dd6640 1 librados: init done 2024-03-25T18:16:28.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.256+0000 7f97c5dd6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:28.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.256+0000 7f97c5dd6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:28.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.256+0000 7f97c5dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:28.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.256+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c004f730 con 0x7f97c00b1b00 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.527+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f97b0001d00 con 0x7f97c00b1b00 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.527+0000 7f97a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.527+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f97b00037b0 con 0x7f97c00b1b00 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.527+0000 7f97a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.527+0000 7f97a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.527+0000 7f97a7fff640 10 monclient: dump: 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.528 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.527+0000 7f97bdffb640 4 set_mon_vals no callback set 2024-03-25T18:16:28.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.528+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f97b00039d0 con 0x7f97c00b1b00 2024-03-25T18:16:28.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.529+0000 7f97beffd640 10 monclient: get_auth_request con 0x7f97ac06ddd0 auth_method 0 2024-03-25T18:16:28.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.529+0000 7f97a7fff640 10 client.4312.objecter ms_handle_connect 0x7f97ac06ddd0 2024-03-25T18:16:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.532+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f97b00534d0 con 0x7f97c00b1b00 2024-03-25T18:16:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.532+0000 7f97a7fff640 10 client.4312.objecter ms_dispatch 0x7f97c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.532+0000 7f97a7fff640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.532+0000 7f97a7fff640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.532+0000 7f97a7fff640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-25T18:16:28.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.533+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f97b004ee10 con 0x7f97c00b1b00 2024-03-25T18:16:28.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.533+0000 7f97a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:28.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.533+0000 7f97a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:28.578 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:35.215 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.596 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:35.233 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.629 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:35.267 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.662 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:35.288 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.665 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:35.292 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.669 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:35.306 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.675 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:35.312 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.690 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:35.316 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.699 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:35.337 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:28.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.741+0000 7f97c5dd6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.741+0000 7f97c5dd6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.741+0000 7f97c5dd6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:16:28.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.741+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97c00aa090 con 0x7f97c00b1b00 2024-03-25T18:16:28.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.864+0000 7f97a7fff640 1 -- 172.21.15.7:0/4275560396 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f97b0050f10 con 0x7f97c00b1b00 2024-03-25T18:16:28.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.864+0000 7f97a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:28.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.864+0000 7f97a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:28.865 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:28.866 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 10 librados: watch_flush enter 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 10 librados: watch_flush exit 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 10 client.4312.objecter successfully canceled tick 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 >> v1:172.21.15.7:6800/1717564767 conn(0x7f97ac06ddd0 legacy=0x7f97ac070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 10 monclient: shutdown 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.867+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 >> v1:172.21.15.183:6789/0 conn(0x7f97c00b1b00 legacy=0x7f97c01358a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.868+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 shutdown_connections 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.868+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 >> 172.21.15.7:0/4275560396 conn(0x7f97c00617a0 msgr2=0x7f97c00b6480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.868+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 shutdown_connections 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.868+0000 7f97c5dd6640 1 -- 172.21.15.7:0/4275560396 wait complete. 2024-03-25T18:16:28.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:28.868+0000 7f97c5dd6640 1 librados: shutdown 2024-03-25T18:16:28.880 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:35.686 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.801+0000 7f399930e640 10 monclient: get_monmap_and_config 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.801+0000 7f399930e640 10 monclient: build_initial_monmap 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.801+0000 7f399930e640 10 monclient: monmap: 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:35.802276+0000 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:35.802276+0000 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:35.802 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:35.803 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:35.803 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:35.803 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:35.803 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.801+0000 7f399930e640 1 Processor -- start 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.801+0000 7f399930e640 1 -- start start 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.801+0000 7f399930e640 10 monclient: init 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 10 monclient(hunting): picked mon.noname-a con 0x7f39940b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 10 monclient(hunting): picked mon.noname-c con 0x7f39940b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:35.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 10 monclient(hunting): picked mon.noname-b con 0x7f399413c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:35.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940ab0e0 con 0x7f39940b4ff0 2024-03-25T18:16:35.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940ab3c0 con 0x7f39940b4670 2024-03-25T18:16:35.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f39940a9f30 con 0x7f399413c480 2024-03-25T18:16:35.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:35.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.802+0000 7f399930e640 10 monclient(hunting): authenticate will time out at 7437.907382s 2024-03-25T18:16:35.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.807+0000 7f3992ffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f39940b4ff0 0x7f399413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:37096/0 (socket says 172.21.15.7:37096) 2024-03-25T18:16:35.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.807+0000 7f3992ffd640 1 -- 172.21.15.7:0/728839727 learned_addr learned my addr 172.21.15.7:0/728839727 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:35.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.818+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2470344975 0 0) 0x7f39940ab0e0 con 0x7f39940b4ff0 2024-03-25T18:16:35.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.818+0000 7f3991ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.818+0000 7f3991ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.818+0000 7f3991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.818+0000 7f3991ffb640 10 monclient(hunting): my global_id is 4320 2024-03-25T18:16:35.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.818+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f397c003520 con 0x7f39940b4ff0 2024-03-25T18:16:35.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.820+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3961195834 0 0) 0x7f39940ab3c0 con 0x7f39940b4670 2024-03-25T18:16:35.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.820+0000 7f3991ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.820+0000 7f3991ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.820+0000 7f3991ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.820+0000 7f3991ffb640 10 monclient(hunting): my global_id is 4315 2024-03-25T18:16:35.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.820+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f39940ab0e0 con 0x7f39940b4670 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3796617792 0 0) 0x7f397c003520 con 0x7f39940b4ff0 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 >> v1:172.21.15.7:6790/0 conn(0x7f399413c480 legacy=0x7f399414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 >> v1:172.21.15.183:6789/0 conn(0x7f39940b4670 legacy=0x7f39941425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 1 monclient: found mon.noname-a 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39940a9b70 con 0x7f39940b4ff0 2024-03-25T18:16:35.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3984001690 con 0x7f39940b4ff0 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f3991ffb640 10 monclient: dump: 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:35.827 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:35.828 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:35.828 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:35.828 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:35.828 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f399930e640 5 monclient: authenticate success, global_id 4320 2024-03-25T18:16:35.828 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.824+0000 7f399930e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f39840017f0 con 0x7f39940b4ff0 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f3991ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f3991ffb640 1 -- 172.21.15.7:0/728839727 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3984001c00 con 0x7f39940b4ff0 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f3991ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f3991ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f3991ffb640 10 monclient: dump: 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:35.843 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f399930e640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:35.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.842+0000 7f399930e640 4 set_mon_vals no callback set 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.843+0000 7f399930e640 10 monclient: shutdown 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.843+0000 7f399930e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.843+0000 7f399930e640 1 -- 172.21.15.7:0/728839727 >> v1:172.21.15.7:6789/0 conn(0x7f39940b4ff0 legacy=0x7f399413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.843+0000 7f399930e640 1 -- 172.21.15.7:0/728839727 shutdown_connections 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.843+0000 7f399930e640 1 -- 172.21.15.7:0/728839727 >> 172.21.15.7:0/728839727 conn(0x7f39940617a0 msgr2=0x7f3994061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.843+0000 7f399930e640 1 -- 172.21.15.7:0/728839727 shutdown_connections 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.843+0000 7f399930e640 1 -- 172.21.15.7:0/728839727 wait complete. 2024-03-25T18:16:35.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 10 monclient: build_initial_monmap 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 10 monclient: monmap: 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 1 librados: starting msgr at 2024-03-25T18:16:35.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 1 librados: starting objecter 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 1 Processor -- start 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 1 -- start start 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 1 librados: setting wanted keys 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 1 librados: calling monclient init 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.844+0000 7f399930e640 10 monclient: init 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 10 monclient(hunting): picked mon.noname-a con 0x7f39940b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 10 monclient(hunting): picked mon.noname-c con 0x7f39940b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 10 monclient(hunting): picked mon.noname-b con 0x7f399413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f399413d0c0 con 0x7f39940b4670 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3994130cc0 con 0x7f399413c480 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3994146010 con 0x7f39940b4ff0 2024-03-25T18:16:35.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:35.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.845+0000 7f399930e640 10 monclient(hunting): authenticate will time out at 7437.950192s 2024-03-25T18:16:35.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.848+0000 7f39937fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f399413c480 0x7f39941e2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:36182/0 (socket says 172.21.15.7:36182) 2024-03-25T18:16:35.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.848+0000 7f39937fe640 1 -- 172.21.15.7:0/4279440436 learned_addr learned my addr 172.21.15.7:0/4279440436 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:35.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.850+0000 7f397b7fe640 10 client.?.objecter ms_handle_connect 0x7f399413c480 2024-03-25T18:16:35.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.850+0000 7f397b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.851+0000 7f397b7fe640 10 client.?.objecter ms_handle_connect 0x7f39940b4ff0 2024-03-25T18:16:35.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.851+0000 7f397b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.859+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137718146 0 0) 0x7f3994130cc0 con 0x7f399413c480 2024-03-25T18:16:35.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.859+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.859+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.859+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.859+0000 7f397b7fe640 10 monclient(hunting): my global_id is 4318 2024-03-25T18:16:35.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.859+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3980004eb0 con 0x7f399413c480 2024-03-25T18:16:35.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.863+0000 7f397b7fe640 10 client.?.objecter ms_handle_connect 0x7f39940b4670 2024-03-25T18:16:35.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.863+0000 7f397b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:35.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.868+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3910612689 0 0) 0x7f3994146010 con 0x7f39940b4ff0 2024-03-25T18:16:35.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.868+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.868+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.868+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.868+0000 7f397b7fe640 10 monclient(hunting): my global_id is 4337 2024-03-25T18:16:35.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.868+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3994130cc0 con 0x7f39940b4ff0 2024-03-25T18:16:35.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.873+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3588826776 0 0) 0x7f3980004eb0 con 0x7f399413c480 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.873+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3994146010 con 0x7f399413c480 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.873+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f398c001c30 con 0x7f399413c480 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.873+0000 7f397b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.873+0000 7f397b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.873+0000 7f397b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.874 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.875 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:35.875 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:35.875 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:35.875 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:35.875 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:35.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.873+0000 7f397b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:35.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.874+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1629824115 0 0) 0x7f399413d0c0 con 0x7f39940b4670 2024-03-25T18:16:35.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.874+0000 7f397b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:16:35.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.874+0000 7f397b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:35.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.874+0000 7f397b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:35.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.874+0000 7f397b7fe640 10 monclient(hunting): my global_id is 4323 2024-03-25T18:16:35.876 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.874+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3980004eb0 con 0x7f39940b4670 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.882+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1971955646 0 0) 0x7f3980004eb0 con 0x7f39940b4670 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.882+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f399413d0c0 con 0x7f39940b4670 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.882+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3984001b00 con 0x7f39940b4670 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.882+0000 7f397b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.882+0000 7f397b7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.882+0000 7f397b7fe640 10 monclient(hunting): dump: 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:35.883 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:35.884 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:35.884 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:35.884 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:35.884 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:35.893 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2095023797 0 0) 0x7f399413d0c0 con 0x7f39940b4670 2024-03-25T18:16:35.893 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 >> v1:172.21.15.7:6790/0 conn(0x7f39940b4ff0 legacy=0x7f39941dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.893 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 >> v1:172.21.15.183:6789/0 conn(0x7f399413c480 legacy=0x7f39941e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:35.893 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 1 monclient: found mon.a 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f399413dbd0 con 0x7f39940b4670 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f397b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 5 monclient: authenticate success, global_id 4323 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 10 monclient: _renew_subs 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f399413deb0 con 0x7f39940b4670 2024-03-25T18:16:35.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 10 client.4323.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:35.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 10 monclient: _renew_subs 2024-03-25T18:16:35.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:35.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f399413e470 con 0x7f39940b4670 2024-03-25T18:16:35.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.892+0000 7f399930e640 1 librados: init done 2024-03-25T18:16:35.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.893+0000 7f399930e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.893+0000 7f399930e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:35.896 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.893+0000 7f399930e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:35.896 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:35.893+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f399406dbd0 con 0x7f39940b4670 2024-03-25T18:16:36.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.222+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3984004a90 con 0x7f39940b4670 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.223+0000 7f397b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.223+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3984001250 con 0x7f39940b4670 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.223+0000 7f397b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.223+0000 7f397b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.223+0000 7f397b7fe640 10 monclient: dump: 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:36.224 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:36.225 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:36.225 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:36.225 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:36.225 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:36.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.223+0000 7f39917fa640 4 set_mon_vals no callback set 2024-03-25T18:16:36.226 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.223+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f398401ac10 con 0x7f39940b4670 2024-03-25T18:16:36.226 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.224+0000 7f39927fc640 10 monclient: get_auth_request con 0x7f398006df20 auth_method 0 2024-03-25T18:16:36.226 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.224+0000 7f397b7fe640 10 client.4323.objecter ms_handle_connect 0x7f398006df20 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f3984054840 con 0x7f39940b4670 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 10 client.4323.objecter ms_dispatch 0x7f3994000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 3 client.4323.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 3 client.4323.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 20 client.4323.objecter dump_active .. 0 homeless 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3984050180 con 0x7f39940b4670 2024-03-25T18:16:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:36.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.232+0000 7f397b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:36.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.494+0000 7f399930e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:36.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.495+0000 7f399930e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:36.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.495+0000 7f399930e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:36.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.495+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f399413d270 con 0x7f39940b4670 2024-03-25T18:16:36.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.524+0000 7f397b7fe640 1 -- 172.21.15.7:0/4279440436 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f399413d270 con 0x7f39940b4670 2024-03-25T18:16:36.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.524+0000 7f397b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:36.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.524+0000 7f397b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:36.526 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:36.526 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:36.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 10 librados: watch_flush enter 2024-03-25T18:16:36.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 10 librados: watch_flush exit 2024-03-25T18:16:36.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 20 client.4323.objecter shutdown clearing up homeless session... 2024-03-25T18:16:36.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 10 client.4323.objecter successfully canceled tick 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 >> v1:172.21.15.7:6800/1717564767 conn(0x7f398006df20 legacy=0x7f39800703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 10 monclient: shutdown 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 >> v1:172.21.15.7:6789/0 conn(0x7f39940b4670 legacy=0x7f39941db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 shutdown_connections 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 >> 172.21.15.7:0/4279440436 conn(0x7f39940617a0 msgr2=0x7f3994140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 shutdown_connections 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 1 -- 172.21.15.7:0/4279440436 wait complete. 2024-03-25T18:16:36.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:36.527+0000 7f399930e640 1 librados: shutdown 2024-03-25T18:16:36.540 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:43.347 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:43.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.460+0000 7f5a50c84640 10 monclient: get_monmap_and_config 2024-03-25T18:16:43.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.460+0000 7f5a50c84640 10 monclient: build_initial_monmap 2024-03-25T18:16:43.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.460+0000 7f5a50c84640 10 monclient: monmap: 2024-03-25T18:16:43.461 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:43.461 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:43.461806+0000 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:43.461806+0000 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:43.462 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:43.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.461+0000 7f5a50c84640 1 Processor -- start 2024-03-25T18:16:43.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.461+0000 7f5a50c84640 1 -- start start 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.461+0000 7f5a50c84640 10 monclient: init 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c0b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 10 monclient(hunting): picked mon.noname-b con 0x7f5a4c0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4c0b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0a96f0 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c0ab0e0 con 0x7f5a4c0b4e70 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c145f70 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:43.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.462+0000 7f5a50c84640 10 monclient(hunting): authenticate will time out at 7445.567262s 2024-03-25T18:16:43.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.466+0000 7f5a4ad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5a4c0b1b00 0x7f5a4c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39532/0 (socket says 172.21.15.7:39532) 2024-03-25T18:16:43.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.466+0000 7f5a4ad9d640 1 -- 172.21.15.7:0/290653664 learned_addr learned my addr 172.21.15.7:0/290653664 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:43.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.477+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 546289270 0 0) 0x7f5a4c0a96f0 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.477+0000 7f5a4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.477+0000 7f5a4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.477+0000 7f5a4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.477+0000 7f5a4959a640 10 monclient(hunting): my global_id is 4326 2024-03-25T18:16:43.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.477+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a34003520 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.484+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 892203414 0 0) 0x7f5a4c0ab0e0 con 0x7f5a4c0b4e70 2024-03-25T18:16:43.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.484+0000 7f5a4959a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.484+0000 7f5a4959a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.484+0000 7f5a4959a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.484+0000 7f5a4959a640 10 monclient(hunting): my global_id is 4321 2024-03-25T18:16:43.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.484+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a4c0a96f0 con 0x7f5a4c0b4e70 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2808942234 0 0) 0x7f5a34003520 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 >> v1:172.21.15.7:6790/0 conn(0x7f5a4c0b44b0 legacy=0x7f5a4c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 >> v1:172.21.15.183:6789/0 conn(0x7f5a4c0b4e70 legacy=0x7f5a4c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 10 monclient: _finish_hunting 0 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 1 monclient: found mon.noname-a 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c0a9b80 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 10 monclient: _finish_auth 0 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 10 monclient: _check_auth_tickets 2024-03-25T18:16:43.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a44001a10 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a50c84640 5 monclient: authenticate success, global_id 4326 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a50c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a4959a640 10 monclient: dump: 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.490 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:43.491 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:43.491 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:43.491 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:43.491 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:43.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.487+0000 7f5a50c84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a44001b70 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a4959a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a4959a640 1 -- 172.21.15.7:0/290653664 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a44001f80 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a4959a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a4959a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a4959a640 10 monclient: dump: 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.496 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.497 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:43.497 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:43.497 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:43.497 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:43.497 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:43.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a50c84640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:43.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.495+0000 7f5a50c84640 4 set_mon_vals no callback set 2024-03-25T18:16:43.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.496+0000 7f5a50c84640 10 monclient: shutdown 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.496+0000 7f5a50c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.496+0000 7f5a50c84640 1 -- 172.21.15.7:0/290653664 >> v1:172.21.15.7:6789/0 conn(0x7f5a4c0b1b00 legacy=0x7f5a4c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.496+0000 7f5a50c84640 1 -- 172.21.15.7:0/290653664 shutdown_connections 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.496+0000 7f5a50c84640 1 -- 172.21.15.7:0/290653664 >> 172.21.15.7:0/290653664 conn(0x7f5a4c0617a0 msgr2=0x7f5a4c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.496+0000 7f5a50c84640 1 -- 172.21.15.7:0/290653664 shutdown_connections 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.496+0000 7f5a50c84640 1 -- 172.21.15.7:0/290653664 wait complete. 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.497+0000 7f5a50c84640 10 monclient: build_initial_monmap 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.497+0000 7f5a50c84640 10 monclient: monmap: 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:43.499 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.498+0000 7f5a50c84640 1 librados: starting msgr at 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.498+0000 7f5a50c84640 1 librados: starting objecter 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.498+0000 7f5a50c84640 1 Processor -- start 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.498+0000 7f5a50c84640 1 -- start start 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.498+0000 7f5a50c84640 1 librados: setting wanted keys 2024-03-25T18:16:43.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.498+0000 7f5a50c84640 1 librados: calling monclient init 2024-03-25T18:16:43.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.498+0000 7f5a50c84640 10 monclient: init 2024-03-25T18:16:43.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:43.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:43.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 10 monclient(hunting): picked mon.noname-b con 0x7f5a4c0b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:43.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 10 monclient(hunting): picked mon.noname-c con 0x7f5a4c0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:43.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 10 monclient(hunting): picked mon.noname-a con 0x7f5a4c0b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:43.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c13c5a0 con 0x7f5a4c0b4e70 2024-03-25T18:16:43.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c13b620 con 0x7f5a4c0b1b00 2024-03-25T18:16:43.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a4c1370d0 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:43.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.499+0000 7f5a50c84640 10 monclient(hunting): authenticate will time out at 7445.603992s 2024-03-25T18:16:43.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.507+0000 7f5a49d9b640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f5a4c0b44b0 0x7f5a4c131e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:47142/0 (socket says 172.21.15.7:47142) 2024-03-25T18:16:43.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.507+0000 7f5a49d9b640 1 -- 172.21.15.7:0/31574787 learned_addr learned my addr 172.21.15.7:0/31574787 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:43.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.509+0000 7f5a3affd640 10 client.?.objecter ms_handle_connect 0x7f5a4c0b4e70 2024-03-25T18:16:43.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.509+0000 7f5a3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.511+0000 7f5a3affd640 10 client.?.objecter ms_handle_connect 0x7f5a4c0b44b0 2024-03-25T18:16:43.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.511+0000 7f5a3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 10 client.?.objecter ms_handle_connect 0x7f5a4c0b1b00 2024-03-25T18:16:43.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:43.522 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889274748 0 0) 0x7f5a4c1370d0 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.522 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 10 monclient(hunting): _negotiate 2024-03-25T18:16:43.522 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:43.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:43.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 10 monclient(hunting): my global_id is 4343 2024-03-25T18:16:43.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.521+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a2c004eb0 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.545+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 110019918 0 0) 0x7f5a2c004eb0 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.545+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a4c1370d0 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.545+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a40001a00 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.545+0000 7f5a3affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.545+0000 7f5a3affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.545+0000 7f5a3affd640 10 monclient(hunting): dump: 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:43.547 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:43.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.545+0000 7f5a3affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:43.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1864215340 0 0) 0x7f5a4c1370d0 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 >> v1:172.21.15.183:6789/0 conn(0x7f5a4c0b1b00 legacy=0x7f5a4c13be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 >> v1:172.21.15.7:6789/0 conn(0x7f5a4c0b4e70 legacy=0x7f5a4c13a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:43.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 10 monclient: _finish_hunting 0 2024-03-25T18:16:43.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 1 monclient: found mon.c 2024-03-25T18:16:43.570 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.568+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a4c137260 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a3affd640 10 monclient: _finish_auth 0 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a3affd640 10 monclient: _check_auth_tickets 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a3affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 5 monclient: authenticate success, global_id 4343 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 10 monclient: _renew_subs 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a4c132570 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 10 monclient: _renew_subs 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a4c13a760 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.569+0000 7f5a50c84640 1 librados: init done 2024-03-25T18:16:43.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.570+0000 7f5a50c84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.570+0000 7f5a50c84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.570+0000 7f5a50c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:43.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.570+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a4c06ec80 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.931+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a40001c80 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.931+0000 7f5a3affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:43.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.931+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a40003a60 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.931+0000 7f5a3affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.931+0000 7f5a3affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.931+0000 7f5a3affd640 10 monclient: dump: 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:43.933 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:43.934 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:43.934 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:43.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.931+0000 7f5a48d99640 4 set_mon_vals no callback set 2024-03-25T18:16:43.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.932+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f5a40003c80 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.933+0000 7f5a4a59c640 10 monclient: get_auth_request con 0x7f5a2c06dd00 auth_method 0 2024-03-25T18:16:43.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.933+0000 7f5a3affd640 10 client.4343.objecter ms_handle_connect 0x7f5a2c06dd00 2024-03-25T18:16:43.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f5a40054830 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 10 client.4343.objecter ms_dispatch 0x7f5a4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:43.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:43.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:43.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-25T18:16:43.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a40050340 con 0x7f5a4c0b44b0 2024-03-25T18:16:43.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:43.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:43.938+0000 7f5a3affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:44.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.200+0000 7f5a50c84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:44.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.200+0000 7f5a50c84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:44.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.200+0000 7f5a50c84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:44.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.200+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a4c04f730 con 0x7f5a4c0b44b0 2024-03-25T18:16:44.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.230+0000 7f5a3affd640 1 -- 172.21.15.7:0/31574787 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f5a40052270 con 0x7f5a4c0b44b0 2024-03-25T18:16:44.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.230+0000 7f5a3affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:44.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.230+0000 7f5a3affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:44.231 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:44.231 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.232+0000 7f5a50c84640 10 librados: watch_flush enter 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 10 librados: watch_flush exit 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 10 client.4343.objecter successfully canceled tick 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 >> v1:172.21.15.7:6800/1717564767 conn(0x7f5a2c06dd00 legacy=0x7f5a2c0701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 10 monclient: shutdown 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:44.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 >> v1:172.21.15.7:6790/0 conn(0x7f5a4c0b44b0 legacy=0x7f5a4c131e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:44.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 shutdown_connections 2024-03-25T18:16:44.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 >> 172.21.15.7:0/31574787 conn(0x7f5a4c0617a0 msgr2=0x7f5a4c14ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:44.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 shutdown_connections 2024-03-25T18:16:44.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 1 -- 172.21.15.7:0/31574787 wait complete. 2024-03-25T18:16:44.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:44.233+0000 7f5a50c84640 1 librados: shutdown 2024-03-25T18:16:44.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:48.364 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:55.001 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.414 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:55.051 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.463 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:02:55.100 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.469 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:55.096 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.479 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:55.105 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.496 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:02:55.122 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.524 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:55.161 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.531 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:55.168 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:48.556 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:02:55.193 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:16:51.050 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:51.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.163+0000 7f2977cc0640 10 monclient: get_monmap_and_config 2024-03-25T18:16:51.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.163+0000 7f2977cc0640 10 monclient: build_initial_monmap 2024-03-25T18:16:51.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.163+0000 7f2977cc0640 10 monclient: monmap: 2024-03-25T18:16:51.164 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:51.164904+0000 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:51.164904+0000 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:51.165 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:51.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.164+0000 7f2977cc0640 1 Processor -- start 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.164+0000 7f2977cc0640 1 -- start start 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.164+0000 7f2977cc0640 10 monclient: init 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f29700b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f29700b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f297013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29700ab0e0 con 0x7f29700b4ff0 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29700ab3c0 con 0x7f29700b4670 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29700a9f30 con 0x7f297013c480 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:51.167 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.165+0000 7f2977cc0640 10 monclient(hunting): authenticate will time out at 7453.270293s 2024-03-25T18:16:51.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.180+0000 7f297525b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f29700b4670 0x7f29701425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:42650/0 (socket says 172.21.15.7:42650) 2024-03-25T18:16:51.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.180+0000 7f297525b640 1 -- 172.21.15.7:0/3126141929 learned_addr learned my addr 172.21.15.7:0/3126141929 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:51.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.193+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 633391516 0 0) 0x7f29700ab3c0 con 0x7f29700b4670 2024-03-25T18:16:51.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.193+0000 7f2974a5a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.193+0000 7f2974a5a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.193+0000 7f2974a5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.193+0000 7f2974a5a640 10 monclient(hunting): my global_id is 4324 2024-03-25T18:16:51.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.193+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f295c003520 con 0x7f29700b4670 2024-03-25T18:16:51.203 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.202+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 115998461 0 0) 0x7f29700ab0e0 con 0x7f29700b4ff0 2024-03-25T18:16:51.203 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.202+0000 7f2974a5a640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.203 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.202+0000 7f2974a5a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.203 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.202+0000 7f2974a5a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.203 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.202+0000 7f2974a5a640 10 monclient(hunting): my global_id is 4329 2024-03-25T18:16:51.203 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.202+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29700ab3c0 con 0x7f29700b4ff0 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2395659025 0 0) 0x7f29700ab3c0 con 0x7f29700b4ff0 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 >> v1:172.21.15.7:6790/0 conn(0x7f297013c480 legacy=0x7f297014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 >> v1:172.21.15.183:6789/0 conn(0x7f29700b4670 legacy=0x7f29701425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 10 monclient: _finish_hunting 0 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 1 monclient: found mon.noname-a 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29700a9b70 con 0x7f29700b4ff0 2024-03-25T18:16:51.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 10 monclient: _finish_auth 0 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 10 monclient: _check_auth_tickets 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f296c001670 con 0x7f29700b4ff0 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2977cc0640 5 monclient: authenticate success, global_id 4329 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2977cc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2974a5a640 10 monclient: dump: 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.210 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:51.211 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:51.211 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:51.211 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:51.211 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:51.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.208+0000 7f2977cc0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f296c0017d0 con 0x7f29700b4ff0 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2974a5a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2974a5a640 1 -- 172.21.15.7:0/3126141929 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f296c001ba0 con 0x7f29700b4ff0 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2977cc0640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2977cc0640 4 set_mon_vals no callback set 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2974a5a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2974a5a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.227+0000 7f2974a5a640 10 monclient: dump: 2024-03-25T18:16:51.228 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:51.229 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:51.230 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.228+0000 7f2977cc0640 10 monclient: shutdown 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.228+0000 7f2977cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.228+0000 7f2977cc0640 1 -- 172.21.15.7:0/3126141929 >> v1:172.21.15.7:6789/0 conn(0x7f29700b4ff0 legacy=0x7f297013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.228+0000 7f2977cc0640 1 -- 172.21.15.7:0/3126141929 shutdown_connections 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.228+0000 7f2977cc0640 1 -- 172.21.15.7:0/3126141929 >> 172.21.15.7:0/3126141929 conn(0x7f29700617a0 msgr2=0x7f2970061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.228+0000 7f2977cc0640 1 -- 172.21.15.7:0/3126141929 shutdown_connections 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.228+0000 7f2977cc0640 1 -- 172.21.15.7:0/3126141929 wait complete. 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 10 monclient: build_initial_monmap 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 10 monclient: monmap: 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:51.231 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 1 librados: starting msgr at 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 1 librados: starting objecter 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 1 Processor -- start 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 1 -- start start 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 1 librados: setting wanted keys 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 1 librados: calling monclient init 2024-03-25T18:16:51.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.229+0000 7f2977cc0640 10 monclient: init 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.230+0000 7f2977cc0640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.230+0000 7f2977cc0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.230+0000 7f2977cc0640 10 monclient(hunting): picked mon.noname-b con 0x7f29700b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.230+0000 7f2977cc0640 10 monclient(hunting): picked mon.noname-a con 0x7f29700b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.231+0000 7f2977cc0640 10 monclient(hunting): picked mon.noname-c con 0x7f297013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.231+0000 7f2977cc0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2970143880 con 0x7f29700b4ff0 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.231+0000 7f2977cc0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2970144890 con 0x7f29700b4670 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.231+0000 7f2977cc0640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2970144ae0 con 0x7f297013c480 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.231+0000 7f2977cc0640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:51.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.231+0000 7f2977cc0640 10 monclient(hunting): authenticate will time out at 7453.335778s 2024-03-25T18:16:51.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.245+0000 7f2975a5c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f29700b4670 0x7f2970145830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:42712/0 (socket says 172.21.15.7:42712) 2024-03-25T18:16:51.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.245+0000 7f2975a5c640 1 -- 172.21.15.7:0/1446068132 learned_addr learned my addr 172.21.15.7:0/1446068132 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:51.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.247+0000 7f29667fc640 10 client.?.objecter ms_handle_connect 0x7f29700b4670 2024-03-25T18:16:51.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.247+0000 7f29667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:51.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.249+0000 7f29667fc640 10 client.?.objecter ms_handle_connect 0x7f297013c480 2024-03-25T18:16:51.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.249+0000 7f29667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:51.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.255+0000 7f29667fc640 10 client.?.objecter ms_handle_connect 0x7f29700b4ff0 2024-03-25T18:16:51.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.255+0000 7f29667fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:51.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.256+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2797932809 0 0) 0x7f2970144ae0 con 0x7f297013c480 2024-03-25T18:16:51.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.256+0000 7f29667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.256+0000 7f29667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.256+0000 7f29667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.256+0000 7f29667fc640 10 monclient(hunting): my global_id is 4349 2024-03-25T18:16:51.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.256+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2954004eb0 con 0x7f297013c480 2024-03-25T18:16:51.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.260+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 155242700 0 0) 0x7f2970144890 con 0x7f29700b4670 2024-03-25T18:16:51.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.260+0000 7f29667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.260+0000 7f29667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.260+0000 7f29667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.260+0000 7f29667fc640 10 monclient(hunting): my global_id is 4327 2024-03-25T18:16:51.261 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.260+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2970144ae0 con 0x7f29700b4670 2024-03-25T18:16:51.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.271+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3821340727 0 0) 0x7f2970143880 con 0x7f29700b4ff0 2024-03-25T18:16:51.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.271+0000 7f29667fc640 10 monclient(hunting): _negotiate 2024-03-25T18:16:51.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.271+0000 7f29667fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:51.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.271+0000 7f29667fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:51.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.271+0000 7f29667fc640 10 monclient(hunting): my global_id is 4332 2024-03-25T18:16:51.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.271+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2970144890 con 0x7f29700b4ff0 2024-03-25T18:16:51.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.278+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4002504522 0 0) 0x7f2954004eb0 con 0x7f297013c480 2024-03-25T18:16:51.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.278+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2970143880 con 0x7f297013c480 2024-03-25T18:16:51.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.278+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2968001b90 con 0x7f297013c480 2024-03-25T18:16:51.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.278+0000 7f29667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.278+0000 7f29667fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:51.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.278+0000 7f29667fc640 10 monclient(hunting): dump: 2024-03-25T18:16:51.279 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:51.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.278+0000 7f29667fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.282+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3232686313 0 0) 0x7f2970144ae0 con 0x7f29700b4670 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.282+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2954004eb0 con 0x7f29700b4670 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.282+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f296c001ba0 con 0x7f29700b4670 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.282+0000 7f29667fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.282+0000 7f29667fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.282+0000 7f29667fc640 10 monclient(hunting): dump: 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.283 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.284 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:51.284 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:51.284 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:51.284 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:51.284 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:51.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 716371557 0 0) 0x7f2970143880 con 0x7f297013c480 2024-03-25T18:16:51.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 >> v1:172.21.15.183:6789/0 conn(0x7f29700b4670 legacy=0x7f2970145830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 >> v1:172.21.15.7:6789/0 conn(0x7f29700b4ff0 legacy=0x7f2970145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 10 monclient: _finish_hunting 0 2024-03-25T18:16:51.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 1 monclient: found mon.c 2024-03-25T18:16:51.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2970144dc0 con 0x7f297013c480 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 10 monclient: _finish_auth 0 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 10 monclient: _check_auth_tickets 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f29667fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 5 monclient: authenticate success, global_id 4349 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 10 monclient: _renew_subs 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297013f1a0 con 0x7f297013c480 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 10 client.4349.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 10 monclient: _renew_subs 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f297013dbe0 con 0x7f297013c480 2024-03-25T18:16:51.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.286+0000 7f2977cc0640 1 librados: init done 2024-03-25T18:16:51.289 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.287+0000 7f2977cc0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.289 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.287+0000 7f2977cc0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.289 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.288+0000 7f2977cc0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:51.289 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.288+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29701435f0 con 0x7f297013c480 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2968002080 con 0x7f297013c480 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f29667fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2968003b10 con 0x7f297013c480 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f29667fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f29667fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f29667fc640 10 monclient: dump: 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:51.628 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:51.629 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:51.629 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:51.629 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:51.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f2967fff640 4 set_mon_vals no callback set 2024-03-25T18:16:51.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.627+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f2968003d30 con 0x7f297013c480 2024-03-25T18:16:51.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.628+0000 7f2975a5c640 10 monclient: get_auth_request con 0x7f295406df20 auth_method 0 2024-03-25T18:16:51.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.628+0000 7f29667fc640 10 client.4349.objecter ms_handle_connect 0x7f295406df20 2024-03-25T18:16:51.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.633+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f29680538b0 con 0x7f297013c480 2024-03-25T18:16:51.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.633+0000 7f29667fc640 10 client.4349.objecter ms_dispatch 0x7f2970000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:51.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.633+0000 7f29667fc640 3 client.4349.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:51.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.633+0000 7f29667fc640 3 client.4349.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:51.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.633+0000 7f29667fc640 20 client.4349.objecter dump_active .. 0 homeless 2024-03-25T18:16:51.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.634+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f296804f3c0 con 0x7f297013c480 2024-03-25T18:16:51.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.634+0000 7f29667fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:51.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.634+0000 7f29667fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:51.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.885+0000 7f2977cc0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.885+0000 7f2977cc0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.885+0000 7f2977cc0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:16:51.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.885+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f297013d940 con 0x7f297013c480 2024-03-25T18:16:51.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.914+0000 7f29667fc640 1 -- 172.21.15.7:0/1446068132 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f29680512f0 con 0x7f297013c480 2024-03-25T18:16:51.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.914+0000 7f29667fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:51.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.914+0000 7f29667fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:51.915 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:51.916 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 10 librados: watch_flush enter 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 10 librados: watch_flush exit 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 20 client.4349.objecter shutdown clearing up homeless session... 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 10 client.4349.objecter successfully canceled tick 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 >> v1:172.21.15.7:6800/1717564767 conn(0x7f295406df20 legacy=0x7f29540703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 10 monclient: shutdown 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.916+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 >> v1:172.21.15.7:6790/0 conn(0x7f297013c480 legacy=0x7f29701e2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.917+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 shutdown_connections 2024-03-25T18:16:51.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.917+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 >> 172.21.15.7:0/1446068132 conn(0x7f29700617a0 msgr2=0x7f2970140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:51.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.917+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 shutdown_connections 2024-03-25T18:16:51.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.917+0000 7f2977cc0640 1 -- 172.21.15.7:0/1446068132 wait complete. 2024-03-25T18:16:51.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:51.917+0000 7f2977cc0640 1 librados: shutdown 2024-03-25T18:16:51.929 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:16:58.736 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:16:58.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.835+0000 7f680381e640 10 monclient: get_monmap_and_config 2024-03-25T18:16:58.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.835+0000 7f680381e640 10 monclient: build_initial_monmap 2024-03-25T18:16:58.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.835+0000 7f680381e640 10 monclient: monmap: 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:16:58.836539+0000 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:16:58.836539+0000 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:58.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.836+0000 7f680381e640 1 Processor -- start 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.836+0000 7f680381e640 1 -- start start 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.836+0000 7f680381e640 10 monclient: init 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 10 monclient(hunting): picked mon.noname-c con 0x7f67fc0b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 10 monclient(hunting): picked mon.noname-b con 0x7f67fc0b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 10 monclient(hunting): picked mon.noname-a con 0x7f67fc13c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0ab0e0 con 0x7f67fc13c480 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0ab3c0 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0a9f30 con 0x7f67fc0b4670 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:58.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.837+0000 7f680381e640 10 monclient(hunting): authenticate will time out at 7460.942164s 2024-03-25T18:16:58.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.853+0000 7f68015ba640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f67fc0b4ff0 0x7f67fc13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:47438/0 (socket says 172.21.15.7:47438) 2024-03-25T18:16:58.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.853+0000 7f68015ba640 1 -- 172.21.15.7:0/3976587040 learned_addr learned my addr 172.21.15.7:0/3976587040 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370403876 0 0) 0x7f67fc0ab0e0 con 0x7f67fc13c480 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): my global_id is 4335 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e8003520 con 0x7f67fc13c480 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798671234 0 0) 0x7f67fc0ab3c0 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 10 monclient(hunting): my global_id is 4330 2024-03-25T18:16:58.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.877+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc0ab0e0 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.883+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1194192781 0 0) 0x7f67fc0a9f30 con 0x7f67fc0b4670 2024-03-25T18:16:58.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.883+0000 7f67f3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.883+0000 7f67f3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.883+0000 7f67f3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.883+0000 7f67f3fff640 10 monclient(hunting): my global_id is 4352 2024-03-25T18:16:58.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.883+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc0ab3c0 con 0x7f67fc0b4670 2024-03-25T18:16:58.896 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2776491364 0 0) 0x7f67fc0ab0e0 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.896 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 >> v1:172.21.15.7:6790/0 conn(0x7f67fc0b4670 legacy=0x7f67fc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.896 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 >> v1:172.21.15.7:6789/0 conn(0x7f67fc13c480 legacy=0x7f67fc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.896 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 1 monclient: found mon.noname-c 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc0a9b70 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 10 monclient: _finish_auth 0 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f67f80015f0 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f680381e640 5 monclient: authenticate success, global_id 4330 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f680381e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:16:58.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f67f3fff640 10 monclient: dump: 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:58.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.895+0000 7f680381e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:16:58.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67f8001750 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f67f3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f67f3fff640 1 -- 172.21.15.7:0/3976587040 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f67f8001b60 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f680381e640 10 monclient: get_monmap_and_config success 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f680381e640 4 set_mon_vals no callback set 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f67f3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f67f3fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.935+0000 7f67f3fff640 10 monclient: dump: 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.936 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.937 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:58.937 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:58.937 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:58.937 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:58.937 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.936+0000 7f680381e640 10 monclient: shutdown 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.936+0000 7f680381e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.936+0000 7f680381e640 1 -- 172.21.15.7:0/3976587040 >> v1:172.21.15.183:6789/0 conn(0x7f67fc0b4ff0 legacy=0x7f67fc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.936+0000 7f680381e640 1 -- 172.21.15.7:0/3976587040 shutdown_connections 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.936+0000 7f680381e640 1 -- 172.21.15.7:0/3976587040 >> 172.21.15.7:0/3976587040 conn(0x7f67fc0617a0 msgr2=0x7f67fc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.936+0000 7f680381e640 1 -- 172.21.15.7:0/3976587040 shutdown_connections 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.936+0000 7f680381e640 1 -- 172.21.15.7:0/3976587040 wait complete. 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 10 monclient: build_initial_monmap 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 10 monclient: monmap: 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:16:58.939 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 1 librados: starting msgr at 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 1 librados: starting objecter 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 1 Processor -- start 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 1 -- start start 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 1 librados: setting wanted keys 2024-03-25T18:16:58.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 1 librados: calling monclient init 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.937+0000 7f680381e640 10 monclient: init 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 10 monclient: _reopen_session rank -1 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 10 monclient(hunting): picked mon.noname-b con 0x7f67fc0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 10 monclient(hunting): picked mon.noname-a con 0x7f67fc0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 10 monclient(hunting): picked mon.noname-c con 0x7f67fc13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0afd20 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0b0000 con 0x7f67fc0b4670 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67fc0b02e0 con 0x7f67fc13c480 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 10 monclient(hunting): _renew_subs 2024-03-25T18:16:58.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.938+0000 7f680381e640 10 monclient(hunting): authenticate will time out at 7461.043097s 2024-03-25T18:16:58.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.944+0000 7f6801dbb640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f67fc13c480 0x7f67fc1e2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:41684/0 (socket says 172.21.15.7:41684) 2024-03-25T18:16:58.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.944+0000 7f6801dbb640 1 -- 172.21.15.7:0/3234343012 learned_addr learned my addr 172.21.15.7:0/3234343012 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:16:58.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.947+0000 7f67f1ffb640 10 client.?.objecter ms_handle_connect 0x7f67fc13c480 2024-03-25T18:16:58.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.947+0000 7f67f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.948+0000 7f67f1ffb640 10 client.?.objecter ms_handle_connect 0x7f67fc0b4ff0 2024-03-25T18:16:58.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.948+0000 7f67f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.949+0000 7f67f1ffb640 10 client.?.objecter ms_handle_connect 0x7f67fc0b4670 2024-03-25T18:16:58.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.949+0000 7f67f1ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:16:58.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.955+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3300271343 0 0) 0x7f67fc0b0000 con 0x7f67fc0b4670 2024-03-25T18:16:58.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.955+0000 7f67f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.955+0000 7f67f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.955+0000 7f67f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.955+0000 7f67f1ffb640 10 monclient(hunting): my global_id is 4333 2024-03-25T18:16:58.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.955+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67e0004eb0 con 0x7f67fc0b4670 2024-03-25T18:16:58.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.960+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2855213897 0 0) 0x7f67fc0b02e0 con 0x7f67fc13c480 2024-03-25T18:16:58.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.960+0000 7f67f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.960+0000 7f67f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.960+0000 7f67f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.960+0000 7f67f1ffb640 10 monclient(hunting): my global_id is 4355 2024-03-25T18:16:58.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.960+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc0b0000 con 0x7f67fc13c480 2024-03-25T18:16:58.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.962+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 821877946 0 0) 0x7f67fc0afd20 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.962+0000 7f67f1ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:16:58.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.962+0000 7f67f1ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:16:58.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.962+0000 7f67f1ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:16:58.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.962+0000 7f67f1ffb640 10 monclient(hunting): my global_id is 4338 2024-03-25T18:16:58.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.962+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67fc0b02e0 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.983+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2684684409 0 0) 0x7f67fc0b02e0 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.983+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67fc0afd20 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.983+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f67ec001830 con 0x7f67fc0b4ff0 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.983+0000 7f67f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.983+0000 7f67f1ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.983+0000 7f67f1ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:58.984 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:58.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.983+0000 7f67f1ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.984+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1900703305 0 0) 0x7f67e0004eb0 con 0x7f67fc0b4670 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.984+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67fc0b02e0 con 0x7f67fc0b4670 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.984+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f67f8001a60 con 0x7f67fc0b4670 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.984+0000 7f67f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.984+0000 7f67f1ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.984+0000 7f67f1ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:58.986 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:58.987 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:58.987 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:58.987 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:58.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.996+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2957388367 0 0) 0x7f67fc0b0000 con 0x7f67fc13c480 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.996+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f67e0004eb0 con 0x7f67fc13c480 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.996+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f67f4001bf0 con 0x7f67fc13c480 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.996+0000 7f67f1ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.996+0000 7f67f1ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:58.996+0000 7f67f1ffb640 10 monclient(hunting): dump: 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:58.997 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 168673086 0 0) 0x7f67fc0afd20 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 >> v1:172.21.15.7:6790/0 conn(0x7f67fc13c480 legacy=0x7f67fc1e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 >> v1:172.21.15.183:6789/0 conn(0x7f67fc0b4670 legacy=0x7f67fc13a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 1 monclient: found mon.a 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67fc0b05c0 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 10 monclient: _finish_auth 0 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f67f1ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 5 monclient: authenticate success, global_id 4338 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 10 monclient: _renew_subs 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67fc0af0b0 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 10 client.4338.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 10 monclient: _renew_subs 2024-03-25T18:16:59.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f67fc0ae1d0 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.002+0000 7f680381e640 1 librados: init done 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.004+0000 7f680381e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.004+0000 7f680381e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.004+0000 7f680381e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:59.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.004+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67fc0688c0 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.006 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.005+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f67ec001d40 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.006 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.005+0000 7f67f1ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:16:59.006 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.005+0000 7f67f37fe640 4 set_mon_vals no callback set 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.024+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f67ec0037f0 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.024+0000 7f67f1ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.024+0000 7f67f1ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.024+0000 7f67f1ffb640 10 monclient: dump: 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:16:59.025 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:16:59.026 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:16:59.026 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:16:59.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.025+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f67ec019840 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.025+0000 7f68015ba640 10 monclient: get_auth_request con 0x7f67e006e0f0 auth_method 0 2024-03-25T18:16:59.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.025+0000 7f67f1ffb640 10 client.4338.objecter ms_handle_connect 0x7f67e006e0f0 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.029+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f67ec0535f0 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.029+0000 7f67f1ffb640 10 client.4338.objecter ms_dispatch 0x7f67fc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.029+0000 7f67f1ffb640 3 client.4338.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.029+0000 7f67f1ffb640 3 client.4338.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:16:59.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.029+0000 7f67f1ffb640 20 client.4338.objecter dump_active .. 0 homeless 2024-03-25T18:16:59.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.336+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f67ec04ef30 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.336+0000 7f67f1ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:16:59.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.336+0000 7f67f1ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:16:59.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.586+0000 7f680381e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.586+0000 7f680381e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.586+0000 7f680381e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:16:59.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.586+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f67fc0af240 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.610+0000 7f67f1ffb640 1 -- 172.21.15.7:0/3234343012 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f67ec051030 con 0x7f67fc0b4ff0 2024-03-25T18:16:59.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.610+0000 7f67f1ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:16:59.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.610+0000 7f67f1ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:16:59.611 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:16:59.611 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:16:59.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 10 librados: watch_flush enter 2024-03-25T18:16:59.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 10 librados: watch_flush exit 2024-03-25T18:16:59.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 20 client.4338.objecter shutdown clearing up homeless session... 2024-03-25T18:16:59.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 10 client.4338.objecter successfully canceled tick 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 >> v1:172.21.15.7:6800/1717564767 conn(0x7f67e006e0f0 legacy=0x7f67e00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 10 monclient: shutdown 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.612+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 >> v1:172.21.15.7:6789/0 conn(0x7f67fc0b4ff0 legacy=0x7f67fc13a800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.613+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 shutdown_connections 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.613+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 >> 172.21.15.7:0/3234343012 conn(0x7f67fc0617a0 msgr2=0x7f67fc145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.613+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 shutdown_connections 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.613+0000 7f680381e640 1 -- 172.21.15.7:0/3234343012 wait complete. 2024-03-25T18:16:59.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:16:59.613+0000 7f680381e640 1 librados: shutdown 2024-03-25T18:16:59.627 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:06.434 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.552+0000 7f8f983c6640 10 monclient: get_monmap_and_config 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.552+0000 7f8f983c6640 10 monclient: build_initial_monmap 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.552+0000 7f8f983c6640 10 monclient: monmap: 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:17:06.553178+0000 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:17:06.553178+0000 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:17:06.553 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:06.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.552+0000 7f8f983c6640 1 Processor -- start 2024-03-25T18:17:06.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.552+0000 7f8f983c6640 1 -- start start 2024-03-25T18:17:06.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.552+0000 7f8f983c6640 10 monclient: init 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.553+0000 7f8f983c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.553+0000 7f8f983c6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.553+0000 7f8f983c6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f900b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.553+0000 7f8f983c6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f900b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.553+0000 7f8f983c6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f9013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.553+0000 7f8f983c6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f900ab0e0 con 0x7f8f9013c480 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.553+0000 7f8f983c6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f900ab3c0 con 0x7f8f900b4670 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.554+0000 7f8f983c6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f900a9f30 con 0x7f8f900b4ff0 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.554+0000 7f8f983c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:06.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.554+0000 7f8f983c6640 10 monclient(hunting): authenticate will time out at 7468.658663s 2024-03-25T18:17:06.569 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.568+0000 7f8f96162640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f8f900b4ff0 0x7f8f9013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:39616/0 (socket says 172.21.15.7:39616) 2024-03-25T18:17:06.569 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.568+0000 7f8f96162640 1 -- 172.21.15.7:0/1008675325 learned_addr learned my addr 172.21.15.7:0/1008675325 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.582+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1310728721 0 0) 0x7f8f900a9f30 con 0x7f8f900b4ff0 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.582+0000 7f8f95160640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.582+0000 7f8f95160640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.582+0000 7f8f95160640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.582+0000 7f8f95160640 10 monclient(hunting): my global_id is 4358 2024-03-25T18:17:06.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.582+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f7c003520 con 0x7f8f900b4ff0 2024-03-25T18:17:06.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.597+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3517254404 0 0) 0x7f8f900ab3c0 con 0x7f8f900b4670 2024-03-25T18:17:06.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.597+0000 7f8f95160640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.597+0000 7f8f95160640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.597+0000 7f8f95160640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.597+0000 7f8f95160640 10 monclient(hunting): my global_id is 4336 2024-03-25T18:17:06.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.597+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f900a9f30 con 0x7f8f900b4670 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 251054074 0 0) 0x7f8f7c003520 con 0x7f8f900b4ff0 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 >> v1:172.21.15.183:6789/0 conn(0x7f8f900b4670 legacy=0x7f8f901425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 >> v1:172.21.15.7:6789/0 conn(0x7f8f9013c480 legacy=0x7f8f9014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 10 monclient: _finish_hunting 0 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 1 monclient: found mon.noname-b 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f900a9b70 con 0x7f8f900b4ff0 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 10 monclient: _finish_auth 0 2024-03-25T18:17:06.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 10 monclient: _check_auth_tickets 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8f8c001690 con 0x7f8f900b4ff0 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f95160640 10 monclient: dump: 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f983c6640 5 monclient: authenticate success, global_id 4358 2024-03-25T18:17:06.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.609+0000 7f8f983c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:06.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f8c0017f0 con 0x7f8f900b4ff0 2024-03-25T18:17:06.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f95160640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f95160640 1 -- 172.21.15.7:0/1008675325 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8f8c001c00 con 0x7f8f900b4ff0 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f983c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f983c6640 4 set_mon_vals no callback set 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f95160640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f95160640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.685+0000 7f8f95160640 10 monclient: dump: 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:06.687 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:06.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.686+0000 7f8f983c6640 10 monclient: shutdown 2024-03-25T18:17:06.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.686+0000 7f8f983c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:06.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.686+0000 7f8f983c6640 1 -- 172.21.15.7:0/1008675325 >> v1:172.21.15.7:6790/0 conn(0x7f8f900b4ff0 legacy=0x7f8f9013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.686+0000 7f8f983c6640 1 -- 172.21.15.7:0/1008675325 shutdown_connections 2024-03-25T18:17:06.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.686+0000 7f8f983c6640 1 -- 172.21.15.7:0/1008675325 >> 172.21.15.7:0/1008675325 conn(0x7f8f900617a0 msgr2=0x7f8f90061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:06.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.686+0000 7f8f983c6640 1 -- 172.21.15.7:0/1008675325 shutdown_connections 2024-03-25T18:17:06.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.686+0000 7f8f983c6640 1 -- 172.21.15.7:0/1008675325 wait complete. 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.687+0000 7f8f983c6640 10 monclient: build_initial_monmap 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.687+0000 7f8f983c6640 10 monclient: monmap: 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.687+0000 7f8f983c6640 1 librados: starting msgr at 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.687+0000 7f8f983c6640 1 librados: starting objecter 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 1 Processor -- start 2024-03-25T18:17:06.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 1 -- start start 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 1 librados: setting wanted keys 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 1 librados: calling monclient init 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient: init 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient(hunting): picked mon.noname-a con 0x7f8f900b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient(hunting): picked mon.noname-c con 0x7f8f900b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient(hunting): picked mon.noname-b con 0x7f8f9013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9013d2f0 con 0x7f8f900b4670 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f90143810 con 0x7f8f9013c480 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f9013e190 con 0x7f8f900b4ff0 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.688+0000 7f8f983c6640 10 monclient(hunting): authenticate will time out at 7468.793245s 2024-03-25T18:17:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.689+0000 7f8f96963640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f8f9013c480 0x7f8f901e2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:45868/0 (socket says 172.21.15.7:45868) 2024-03-25T18:17:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.689+0000 7f8f96963640 1 -- 172.21.15.7:0/320981215 learned_addr learned my addr 172.21.15.7:0/320981215 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.689+0000 7f8f86ffd640 10 client.?.objecter ms_handle_connect 0x7f8f9013c480 2024-03-25T18:17:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.689+0000 7f8f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.710+0000 7f8f86ffd640 10 client.?.objecter ms_handle_connect 0x7f8f900b4ff0 2024-03-25T18:17:06.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.710+0000 7f8f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.729+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3867778035 0 0) 0x7f8f9013e190 con 0x7f8f900b4ff0 2024-03-25T18:17:06.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.729+0000 7f8f86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.729+0000 7f8f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.729+0000 7f8f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.729+0000 7f8f86ffd640 10 monclient(hunting): my global_id is 4361 2024-03-25T18:17:06.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.729+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f74004eb0 con 0x7f8f900b4ff0 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.743+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 998013977 0 0) 0x7f8f74004eb0 con 0x7f8f900b4ff0 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.743+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8f9013e190 con 0x7f8f900b4ff0 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.743+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8f80001930 con 0x7f8f900b4ff0 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.743+0000 7f8f86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.743+0000 7f8f86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.743+0000 7f8f86ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:06.744 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:06.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.743+0000 7f8f86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.752+0000 7f8f86ffd640 10 client.?.objecter ms_handle_connect 0x7f8f900b4670 2024-03-25T18:17:06.753 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.752+0000 7f8f86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.753+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 147558956 0 0) 0x7f8f90143810 con 0x7f8f9013c480 2024-03-25T18:17:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.753+0000 7f8f86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.753+0000 7f8f86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.753+0000 7f8f86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.753+0000 7f8f86ffd640 10 monclient(hunting): my global_id is 4339 2024-03-25T18:17:06.754 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.753+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f74004eb0 con 0x7f8f9013c480 2024-03-25T18:17:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3494248609 0 0) 0x7f8f9013e190 con 0x7f8f900b4ff0 2024-03-25T18:17:06.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 >> v1:172.21.15.183:6789/0 conn(0x7f8f9013c480 legacy=0x7f8f901e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 >> v1:172.21.15.7:6789/0 conn(0x7f8f900b4670 legacy=0x7f8f9013cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 1 monclient: found mon.c 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f9013e470 con 0x7f8f900b4ff0 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 5 monclient: authenticate success, global_id 4361 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 10 monclient: _renew_subs 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f90145e80 con 0x7f8f900b4ff0 2024-03-25T18:17:06.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 10 client.4361.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 10 monclient: _renew_subs 2024-03-25T18:17:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f9013f800 con 0x7f8f900b4ff0 2024-03-25T18:17:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.759+0000 7f8f983c6640 1 librados: init done 2024-03-25T18:17:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.760+0000 7f8f983c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.760+0000 7f8f983c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.760+0000 7f8f983c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:06.763 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:06.760+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f9013e890 con 0x7f8f900b4ff0 2024-03-25T18:17:07.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.037+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8f80003610 con 0x7f8f900b4ff0 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.037+0000 7f8f86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.037+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8f80003880 con 0x7f8f900b4ff0 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.037+0000 7f8f86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.037+0000 7f8f86ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.037+0000 7f8f86ffd640 10 monclient: dump: 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:07.039 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.037+0000 7f8f9495f640 4 set_mon_vals no callback set 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.038+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f8f80003aa0 con 0x7f8f900b4ff0 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.039+0000 7f8f96162640 10 monclient: get_auth_request con 0x7f8f7406de20 auth_method 0 2024-03-25T18:17:07.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.039+0000 7f8f86ffd640 10 client.4361.objecter ms_handle_connect 0x7f8f7406de20 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f8f80053480 con 0x7f8f900b4ff0 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 10 client.4361.objecter ms_dispatch 0x7f8f90000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 3 client.4361.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 3 client.4361.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 20 client.4361.objecter dump_active .. 0 homeless 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8f8004ef90 con 0x7f8f900b4ff0 2024-03-25T18:17:07.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:07.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.048+0000 7f8f86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:07.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.278+0000 7f8f983c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:07.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.278+0000 7f8f983c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:07.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.278+0000 7f8f983c6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:07.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.278+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8f9006dbd0 con 0x7f8f900b4ff0 2024-03-25T18:17:07.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.324+0000 7f8f86ffd640 1 -- 172.21.15.7:0/320981215 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f8f80050ec0 con 0x7f8f900b4ff0 2024-03-25T18:17:07.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.324+0000 7f8f86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:07.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.324+0000 7f8f86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:07.325 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:17:07.325 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:07.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 10 librados: watch_flush enter 2024-03-25T18:17:07.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 10 librados: watch_flush exit 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 20 client.4361.objecter shutdown clearing up homeless session... 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 10 client.4361.objecter successfully canceled tick 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 >> v1:172.21.15.7:6800/1717564767 conn(0x7f8f7406de20 legacy=0x7f8f740702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 10 monclient: shutdown 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 >> v1:172.21.15.7:6790/0 conn(0x7f8f900b4ff0 legacy=0x7f8f9013ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 shutdown_connections 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 >> 172.21.15.7:0/320981215 conn(0x7f8f900617a0 msgr2=0x7f8f901407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 shutdown_connections 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 1 -- 172.21.15.7:0/320981215 wait complete. 2024-03-25T18:17:07.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:07.326+0000 7f8f983c6640 1 librados: shutdown 2024-03-25T18:17:07.339 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:08.466 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:15.103 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.479 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:15.116 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.511 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:15.148 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.524 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:15.161 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.528 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:15.154 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.534 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:15.172 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.535 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:15.162 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.562 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:15.199 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:08.570 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:15.196 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:14.145 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:14.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.262+0000 7fdd5873a640 10 monclient: get_monmap_and_config 2024-03-25T18:17:14.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.262+0000 7fdd5873a640 10 monclient: build_initial_monmap 2024-03-25T18:17:14.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.262+0000 7fdd5873a640 10 monclient: monmap: 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:17:14.263630+0000 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:17:14.263630+0000 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:17:14.264 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:14.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.263+0000 7fdd5873a640 1 Processor -- start 2024-03-25T18:17:14.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.263+0000 7fdd5873a640 1 -- start start 2024-03-25T18:17:14.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.263+0000 7fdd5873a640 10 monclient: init 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 10 monclient(hunting): picked mon.noname-a con 0x7fdd500b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 10 monclient(hunting): picked mon.noname-b con 0x7fdd500b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 10 monclient(hunting): picked mon.noname-c con 0x7fdd5013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500ab0e0 con 0x7fdd500b4ff0 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500ab3c0 con 0x7fdd5013c480 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd500a9f30 con 0x7fdd500b4670 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:14.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.264+0000 7fdd5873a640 10 monclient(hunting): authenticate will time out at 7476.369056s 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.275+0000 7fdd56cd7640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdd5013c480 0x7fdd5014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:38770/0 (socket says 172.21.15.7:38770) 2024-03-25T18:17:14.276 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.275+0000 7fdd56cd7640 1 -- 172.21.15.7:0/3149417758 learned_addr learned my addr 172.21.15.7:0/3149417758 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:14.290 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.290+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 838769187 0 0) 0x7fdd500ab0e0 con 0x7fdd500b4ff0 2024-03-25T18:17:14.290 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.290+0000 7fdd554d4640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.291 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.290+0000 7fdd554d4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.291 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.290+0000 7fdd554d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.291 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.290+0000 7fdd554d4640 10 monclient(hunting): my global_id is 4344 2024-03-25T18:17:14.291 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.290+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd3c003520 con 0x7fdd500b4ff0 2024-03-25T18:17:14.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.291+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828598843 0 0) 0x7fdd500ab3c0 con 0x7fdd5013c480 2024-03-25T18:17:14.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.291+0000 7fdd554d4640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.291+0000 7fdd554d4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.291+0000 7fdd554d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.291+0000 7fdd554d4640 10 monclient(hunting): my global_id is 4342 2024-03-25T18:17:14.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.291+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd500ab0e0 con 0x7fdd5013c480 2024-03-25T18:17:14.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.292+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3303907100 0 0) 0x7fdd500a9f30 con 0x7fdd500b4670 2024-03-25T18:17:14.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.292+0000 7fdd554d4640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.292+0000 7fdd554d4640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.292+0000 7fdd554d4640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.292+0000 7fdd554d4640 10 monclient(hunting): my global_id is 4364 2024-03-25T18:17:14.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.292+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd500ab3c0 con 0x7fdd500b4670 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4113818879 0 0) 0x7fdd500ab0e0 con 0x7fdd5013c480 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 >> v1:172.21.15.7:6790/0 conn(0x7fdd500b4670 legacy=0x7fdd501425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 >> v1:172.21.15.7:6789/0 conn(0x7fdd500b4ff0 legacy=0x7fdd5013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 10 monclient: _finish_hunting 0 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 1 monclient: found mon.noname-c 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd500a9b70 con 0x7fdd5013c480 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 10 monclient: _finish_auth 0 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 10 monclient: _check_auth_tickets 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdd480019f0 con 0x7fdd5013c480 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:14.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd554d4640 10 monclient: dump: 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd5873a640 5 monclient: authenticate success, global_id 4342 2024-03-25T18:17:14.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.301+0000 7fdd5873a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd48001b50 con 0x7fdd5013c480 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd554d4640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd554d4640 1 -- 172.21.15.7:0/3149417758 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdd48001f60 con 0x7fdd5013c480 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd5873a640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:14.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd5873a640 4 set_mon_vals no callback set 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd554d4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd554d4640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.313+0000 7fdd554d4640 10 monclient: dump: 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:14.315 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.314+0000 7fdd5873a640 10 monclient: shutdown 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.314+0000 7fdd5873a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.314+0000 7fdd5873a640 1 -- 172.21.15.7:0/3149417758 >> v1:172.21.15.183:6789/0 conn(0x7fdd5013c480 legacy=0x7fdd5014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.314+0000 7fdd5873a640 1 -- 172.21.15.7:0/3149417758 shutdown_connections 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.314+0000 7fdd5873a640 1 -- 172.21.15.7:0/3149417758 >> 172.21.15.7:0/3149417758 conn(0x7fdd500617a0 msgr2=0x7fdd50061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.314+0000 7fdd5873a640 1 -- 172.21.15.7:0/3149417758 shutdown_connections 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.314+0000 7fdd5873a640 1 -- 172.21.15.7:0/3149417758 wait complete. 2024-03-25T18:17:14.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.315+0000 7fdd5873a640 10 monclient: build_initial_monmap 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.315+0000 7fdd5873a640 10 monclient: monmap: 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 1 librados: starting msgr at 2024-03-25T18:17:14.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 1 librados: starting objecter 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 1 Processor -- start 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 1 -- start start 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 1 librados: setting wanted keys 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 1 librados: calling monclient init 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 10 monclient: init 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 10 monclient(hunting): picked mon.noname-a con 0x7fdd500b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 10 monclient(hunting): picked mon.noname-c con 0x7fdd500b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 10 monclient(hunting): picked mon.noname-b con 0x7fdd5013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:14.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.316+0000 7fdd5873a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd50144720 con 0x7fdd500b4670 2024-03-25T18:17:14.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.317+0000 7fdd5873a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5013eb10 con 0x7fdd5013c480 2024-03-25T18:17:14.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.317+0000 7fdd5873a640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd5013f1a0 con 0x7fdd500b4ff0 2024-03-25T18:17:14.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.317+0000 7fdd5873a640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:14.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.317+0000 7fdd5873a640 10 monclient(hunting): authenticate will time out at 7476.421689s 2024-03-25T18:17:14.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.318+0000 7fdd564d6640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fdd500b4670 0x7fdd50145190 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:53026/0 (socket says 172.21.15.7:53026) 2024-03-25T18:17:14.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.318+0000 7fdd564d6640 1 -- 172.21.15.7:0/3159188824 learned_addr learned my addr 172.21.15.7:0/3159188824 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:14.321 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.320+0000 7fdd46ffd640 10 client.?.objecter ms_handle_connect 0x7fdd500b4670 2024-03-25T18:17:14.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.320+0000 7fdd46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:14.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.322+0000 7fdd46ffd640 10 client.?.objecter ms_handle_connect 0x7fdd500b4ff0 2024-03-25T18:17:14.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.322+0000 7fdd46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.327+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 475636651 0 0) 0x7fdd50144720 con 0x7fdd500b4670 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.327+0000 7fdd46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.327+0000 7fdd46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.327+0000 7fdd46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.327+0000 7fdd46ffd640 10 monclient(hunting): my global_id is 4347 2024-03-25T18:17:14.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.327+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34004eb0 con 0x7fdd500b4670 2024-03-25T18:17:14.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.333+0000 7fdd46ffd640 10 client.?.objecter ms_handle_connect 0x7fdd5013c480 2024-03-25T18:17:14.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.333+0000 7fdd46ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 939696912 0 0) 0x7fdd5013eb10 con 0x7fdd5013c480 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): my global_id is 4345 2024-03-25T18:17:14.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd50144720 con 0x7fdd5013c480 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2473865471 0 0) 0x7fdd34004eb0 con 0x7fdd500b4670 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd5013eb10 con 0x7fdd500b4670 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdd4c001990 con 0x7fdd500b4670 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:14.343 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:14.344 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:14.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.341+0000 7fdd46ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:14.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.343+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1312241277 0 0) 0x7fdd5013f1a0 con 0x7fdd500b4ff0 2024-03-25T18:17:14.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.343+0000 7fdd46ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:14.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.343+0000 7fdd46ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:14.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.343+0000 7fdd46ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:14.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.343+0000 7fdd46ffd640 10 monclient(hunting): my global_id is 4367 2024-03-25T18:17:14.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.343+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd34004eb0 con 0x7fdd500b4ff0 2024-03-25T18:17:14.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.349+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2051679621 0 0) 0x7fdd50144720 con 0x7fdd5013c480 2024-03-25T18:17:14.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.349+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd5013f1a0 con 0x7fdd5013c480 2024-03-25T18:17:14.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.349+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdd48001e60 con 0x7fdd5013c480 2024-03-25T18:17:14.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.349+0000 7fdd46ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.349+0000 7fdd46ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:14.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.349+0000 7fdd46ffd640 10 monclient(hunting): dump: 2024-03-25T18:17:14.350 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:14.351 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:14.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.350+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4120507820 0 0) 0x7fdd34004eb0 con 0x7fdd500b4ff0 2024-03-25T18:17:14.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.350+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdd50144720 con 0x7fdd500b4ff0 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4192225789 0 0) 0x7fdd5013eb10 con 0x7fdd500b4670 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 >> v1:172.21.15.7:6790/0 conn(0x7fdd500b4ff0 legacy=0x7fdd5013c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 >> v1:172.21.15.183:6789/0 conn(0x7fdd5013c480 legacy=0x7fdd5013d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 1 monclient: found mon.a 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.351+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd50146650 con 0x7fdd500b4670 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd46ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd46ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd46ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:14.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 5 monclient: authenticate success, global_id 4347 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 10 monclient: _renew_subs 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd5013d730 con 0x7fdd500b4670 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 10 client.4347.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 10 monclient: _renew_subs 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd501437a0 con 0x7fdd500b4670 2024-03-25T18:17:14.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.352+0000 7fdd5873a640 1 librados: init done 2024-03-25T18:17:14.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.353+0000 7fdd5873a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:14.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.353+0000 7fdd5873a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:14.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.353+0000 7fdd5873a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:14.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.353+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd50143410 con 0x7fdd500b4670 2024-03-25T18:17:14.654 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.653+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdd4c003610 con 0x7fdd500b4670 2024-03-25T18:17:14.654 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.653+0000 7fdd46ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.653+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdd4c0039e0 con 0x7fdd500b4670 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.653+0000 7fdd46ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.653+0000 7fdd46ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.653+0000 7fdd46ffd640 10 monclient: dump: 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:14.655 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:14.656 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:14.656 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:14.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.653+0000 7fdd54cd3640 4 set_mon_vals no callback set 2024-03-25T18:17:14.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.654+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fdd4c003c00 con 0x7fdd500b4670 2024-03-25T18:17:14.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.655+0000 7fdd55cd5640 10 monclient: get_auth_request con 0x7fdd3406e260 auth_method 0 2024-03-25T18:17:14.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.655+0000 7fdd46ffd640 10 client.4347.objecter ms_handle_connect 0x7fdd3406e260 2024-03-25T18:17:14.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fdd4c0547a0 con 0x7fdd500b4670 2024-03-25T18:17:14.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 10 client.4347.objecter ms_dispatch 0x7fdd50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:14.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 3 client.4347.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:14.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 3 client.4347.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:14.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 20 client.4347.objecter dump_active .. 0 homeless 2024-03-25T18:17:14.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdd4c0502b0 con 0x7fdd500b4670 2024-03-25T18:17:14.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:14.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.660+0000 7fdd46ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:14.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.882+0000 7fdd5873a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.882+0000 7fdd5873a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.882+0000 7fdd5873a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:14.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.882+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdd501467e0 con 0x7fdd500b4670 2024-03-25T18:17:14.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.954+0000 7fdd46ffd640 1 -- 172.21.15.7:0/3159188824 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fdd4c0521e0 con 0x7fdd500b4670 2024-03-25T18:17:14.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.954+0000 7fdd46ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:14.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.954+0000 7fdd46ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:14.955 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:17:14.955 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.956+0000 7fdd5873a640 10 librados: watch_flush enter 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.956+0000 7fdd5873a640 10 librados: watch_flush exit 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.956+0000 7fdd5873a640 20 client.4347.objecter shutdown clearing up homeless session... 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.956+0000 7fdd5873a640 10 client.4347.objecter successfully canceled tick 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 >> v1:172.21.15.7:6800/1717564767 conn(0x7fdd3406e260 legacy=0x7fdd34070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 10 monclient: shutdown 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 >> v1:172.21.15.7:6789/0 conn(0x7fdd500b4670 legacy=0x7fdd50145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:14.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 shutdown_connections 2024-03-25T18:17:14.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 >> 172.21.15.7:0/3159188824 conn(0x7fdd500617a0 msgr2=0x7fdd501405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:14.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 shutdown_connections 2024-03-25T18:17:14.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 1 -- 172.21.15.7:0/3159188824 wait complete. 2024-03-25T18:17:14.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:14.957+0000 7fdd5873a640 1 librados: shutdown 2024-03-25T18:17:14.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:21.777 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.896+0000 7f423c1af640 10 monclient: get_monmap_and_config 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.896+0000 7f423c1af640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.896+0000 7f423c1af640 10 monclient: monmap: 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:17:21.897202+0000 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:17:21.897202+0000 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:17:21.897 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:17:21.898 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.896+0000 7f423c1af640 1 Processor -- start 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.897+0000 7f423c1af640 1 -- start start 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.897+0000 7f423c1af640 10 monclient: init 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.897+0000 7f423c1af640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.897+0000 7f423c1af640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.897+0000 7f423c1af640 10 monclient(hunting): picked mon.noname-c con 0x7f42340b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.897+0000 7f423c1af640 10 monclient(hunting): picked mon.noname-a con 0x7f42340b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.898+0000 7f423c1af640 10 monclient(hunting): picked mon.noname-b con 0x7f42340b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.898+0000 7f423c1af640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42340a96f0 con 0x7f42340b44b0 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.898+0000 7f423c1af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42340ab0e0 con 0x7f42340b4e70 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.898+0000 7f423c1af640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234145f70 con 0x7f42340b1b00 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.898+0000 7f423c1af640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.898+0000 7f423c1af640 10 monclient(hunting): authenticate will time out at 7484.002798s 2024-03-25T18:17:21.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.900+0000 7f423974a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f42340b44b0 0x7f4234145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35364/0 (socket says 172.21.15.7:35364) 2024-03-25T18:17:21.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.900+0000 7f423974a640 1 -- 172.21.15.7:0/1292564249 learned_addr learned my addr 172.21.15.7:0/1292564249 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:21.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.915+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751353688 0 0) 0x7f42340a96f0 con 0x7f42340b44b0 2024-03-25T18:17:21.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.915+0000 7f4238f49640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.915+0000 7f4238f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.915+0000 7f4238f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.915+0000 7f4238f49640 10 monclient(hunting): my global_id is 4350 2024-03-25T18:17:21.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.915+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220003520 con 0x7f42340b44b0 2024-03-25T18:17:21.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.925+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 559444525 0 0) 0x7f42340ab0e0 con 0x7f42340b4e70 2024-03-25T18:17:21.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.925+0000 7f4238f49640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.925+0000 7f4238f49640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.925+0000 7f4238f49640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.925+0000 7f4238f49640 10 monclient(hunting): my global_id is 4348 2024-03-25T18:17:21.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.925+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42340a96f0 con 0x7f42340b4e70 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.927+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 176639900 0 0) 0x7f4220003520 con 0x7f42340b44b0 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 >> v1:172.21.15.7:6790/0 conn(0x7f42340b1b00 legacy=0x7f423414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 >> v1:172.21.15.183:6789/0 conn(0x7f42340b4e70 legacy=0x7f423413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 10 monclient: _finish_hunting 0 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 1 monclient: found mon.noname-a 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42340a9b80 con 0x7f42340b44b0 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 10 monclient: _finish_auth 0 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 10 monclient: _check_auth_tickets 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4224001690 con 0x7f42340b44b0 2024-03-25T18:17:21.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f423c1af640 5 monclient: authenticate success, global_id 4350 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f423c1af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f4238f49640 10 monclient: dump: 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:21.930 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:21.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.928+0000 7f423c1af640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42240017f0 con 0x7f42340b44b0 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f4238f49640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f4238f49640 1 -- 172.21.15.7:0/1292564249 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4224001c00 con 0x7f42340b44b0 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f423c1af640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f423c1af640 4 set_mon_vals no callback set 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f4238f49640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f4238f49640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.935+0000 7f4238f49640 10 monclient: dump: 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.936 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.937 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:21.937 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:21.937 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:21.937 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:21.937 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.936+0000 7f423c1af640 10 monclient: shutdown 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.936+0000 7f423c1af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.936+0000 7f423c1af640 1 -- 172.21.15.7:0/1292564249 >> v1:172.21.15.7:6789/0 conn(0x7f42340b44b0 legacy=0x7f4234145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.936+0000 7f423c1af640 1 -- 172.21.15.7:0/1292564249 shutdown_connections 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.936+0000 7f423c1af640 1 -- 172.21.15.7:0/1292564249 >> 172.21.15.7:0/1292564249 conn(0x7f42340617a0 msgr2=0x7f4234061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.936+0000 7f423c1af640 1 -- 172.21.15.7:0/1292564249 shutdown_connections 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.936+0000 7f423c1af640 1 -- 172.21.15.7:0/1292564249 wait complete. 2024-03-25T18:17:21.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 10 monclient: build_initial_monmap 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 10 monclient: monmap: 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 1 librados: starting msgr at 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 1 librados: starting objecter 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 1 Processor -- start 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 1 -- start start 2024-03-25T18:17:21.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 1 librados: setting wanted keys 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 1 librados: calling monclient init 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.937+0000 7f423c1af640 10 monclient: init 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 10 monclient(hunting): picked mon.noname-c con 0x7f42340b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 10 monclient(hunting): picked mon.noname-a con 0x7f42340b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 10 monclient(hunting): picked mon.noname-b con 0x7f42340b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234142940 con 0x7f42340b44b0 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234142bb0 con 0x7f42340b4e70 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4234140c40 con 0x7f42340b1b00 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:21.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.938+0000 7f423c1af640 10 monclient(hunting): authenticate will time out at 7484.042767s 2024-03-25T18:17:21.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.949+0000 7f423974a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f42340b44b0 0x7f42341df5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35384/0 (socket says 172.21.15.7:35384) 2024-03-25T18:17:21.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.949+0000 7f423974a640 1 -- 172.21.15.7:0/2769812003 learned_addr learned my addr 172.21.15.7:0/2769812003 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:21.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.949+0000 7f422a7fc640 10 client.?.objecter ms_handle_connect 0x7f42340b44b0 2024-03-25T18:17:21.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.949+0000 7f422a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.957+0000 7f422a7fc640 10 client.?.objecter ms_handle_connect 0x7f42340b1b00 2024-03-25T18:17:21.958 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.957+0000 7f422a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.961+0000 7f422a7fc640 10 client.?.objecter ms_handle_connect 0x7f42340b4e70 2024-03-25T18:17:21.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.961+0000 7f422a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:21.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.963+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 278636449 0 0) 0x7f4234140c40 con 0x7f42340b1b00 2024-03-25T18:17:21.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.963+0000 7f422a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.963+0000 7f422a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.963+0000 7f422a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.963+0000 7f422a7fc640 10 monclient(hunting): my global_id is 4373 2024-03-25T18:17:21.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.963+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4218004eb0 con 0x7f42340b1b00 2024-03-25T18:17:21.966 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.965+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3069794096 0 0) 0x7f4234142940 con 0x7f42340b44b0 2024-03-25T18:17:21.966 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.966+0000 7f422a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.966 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.966+0000 7f422a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.966+0000 7f422a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.966+0000 7f422a7fc640 10 monclient(hunting): my global_id is 4353 2024-03-25T18:17:21.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.966+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234140c40 con 0x7f42340b44b0 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.973+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1156741695 0 0) 0x7f4234140c40 con 0x7f42340b44b0 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.973+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4234142940 con 0x7f42340b44b0 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.973+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4224001b40 con 0x7f42340b44b0 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.973+0000 7f422a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.973+0000 7f422a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.973+0000 7f422a7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.974 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.975 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.975 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:21.975 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:21.975 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:21.975 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:21.975 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:21.975 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.973+0000 7f422a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4101245729 0 0) 0x7f4234142bb0 con 0x7f42340b4e70 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 10 monclient(hunting): my global_id is 4351 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4234140c40 con 0x7f42340b4e70 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2905761067 0 0) 0x7f4218004eb0 con 0x7f42340b1b00 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4234142bb0 con 0x7f42340b1b00 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f42300018b0 con 0x7f42340b1b00 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.974+0000 7f422a7fc640 10 monclient(hunting): dump: 2024-03-25T18:17:21.976 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:21.977 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2304365204 0 0) 0x7f4234142bb0 con 0x7f42340b1b00 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 >> v1:172.21.15.183:6789/0 conn(0x7f42340b4e70 legacy=0x7f42341e2d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 >> v1:172.21.15.7:6789/0 conn(0x7f42340b44b0 legacy=0x7f42341df5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 1 monclient: found mon.c 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4234141aa0 con 0x7f42340b1b00 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:21.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f422a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 5 monclient: authenticate success, global_id 4373 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 10 monclient: _renew_subs 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4234133760 con 0x7f42340b1b00 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 10 monclient: _renew_subs 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f42341431b0 con 0x7f42340b1b00 2024-03-25T18:17:21.988 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.986+0000 7f423c1af640 1 librados: init done 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.987+0000 7f423c1af640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.987+0000 7f423c1af640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.987+0000 7f423c1af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:21.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:21.987+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4234140aa0 con 0x7f42340b1b00 2024-03-25T18:17:22.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f42300035f0 con 0x7f42340b1b00 2024-03-25T18:17:22.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:22.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4230003760 con 0x7f42340b1b00 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422a7fc640 10 monclient: dump: 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:22.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422bfff640 4 set_mon_vals no callback set 2024-03-25T18:17:22.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.300+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f4230019730 con 0x7f42340b1b00 2024-03-25T18:17:22.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.301+0000 7f423974a640 10 monclient: get_auth_request con 0x7f421806df20 auth_method 0 2024-03-25T18:17:22.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.301+0000 7f422a7fc640 10 client.4373.objecter ms_handle_connect 0x7f421806df20 2024-03-25T18:17:22.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f4230053420 con 0x7f42340b1b00 2024-03-25T18:17:22.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 10 client.4373.objecter ms_dispatch 0x7f4234000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:22.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:22.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:22.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-25T18:17:22.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f423004ed60 con 0x7f42340b1b00 2024-03-25T18:17:22.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:22.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.307+0000 7f422a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:22.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.571+0000 7f423c1af640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:22.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.571+0000 7f423c1af640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:22.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.571+0000 7f423c1af640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:22.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.571+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f423404f730 con 0x7f42340b1b00 2024-03-25T18:17:22.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.600+0000 7f422a7fc640 1 -- 172.21.15.7:0/2769812003 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f4230050e60 con 0x7f42340b1b00 2024-03-25T18:17:22.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.600+0000 7f422a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:22.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.600+0000 7f422a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:22.601 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:17:22.601 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 10 librados: watch_flush enter 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 10 librados: watch_flush exit 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 10 client.4373.objecter successfully canceled tick 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 >> v1:172.21.15.7:6800/1717564767 conn(0x7f421806df20 legacy=0x7f42180703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 10 monclient: shutdown 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 >> v1:172.21.15.7:6790/0 conn(0x7f42340b1b00 legacy=0x7f4234143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 shutdown_connections 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 >> 172.21.15.7:0/2769812003 conn(0x7f42340617a0 msgr2=0x7f42340b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 shutdown_connections 2024-03-25T18:17:22.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.602+0000 7f423c1af640 1 -- 172.21.15.7:0/2769812003 wait complete. 2024-03-25T18:17:22.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:22.603+0000 7f423c1af640 1 librados: shutdown 2024-03-25T18:17:22.613 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:28.527 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:35.163 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.536 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:35.173 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.575 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:35.213 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.581 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:35.207 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.590 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:35.216 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.598 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:35.235 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.604 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:35.241 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.611 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:35.237 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:28.627 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:35.265 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:29.420 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:29.535 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.534+0000 7f983cf10640 10 monclient: get_monmap_and_config 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.534+0000 7f983cf10640 10 monclient: build_initial_monmap 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.534+0000 7f983cf10640 10 monclient: monmap: 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:17:29.535897+0000 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:17:29.535897+0000 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:17:29.536 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:29.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.535+0000 7f983cf10640 1 Processor -- start 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.535+0000 7f983cf10640 1 -- start start 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.535+0000 7f983cf10640 10 monclient: init 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 10 monclient(hunting): picked mon.noname-c con 0x7f98380b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 10 monclient(hunting): picked mon.noname-a con 0x7f98380b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 10 monclient(hunting): picked mon.noname-b con 0x7f983813c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380ab0e0 con 0x7f98380b4670 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380ab3c0 con 0x7f98380b4ff0 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98380a9f30 con 0x7f983813c480 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:29.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.536+0000 7f983cf10640 10 monclient(hunting): authenticate will time out at 7491.641567s 2024-03-25T18:17:29.541 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.541+0000 7f982ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f98380b4670 0x7f98381425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40156/0 (socket says 172.21.15.7:40156) 2024-03-25T18:17:29.541 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.541+0000 7f982ffff640 1 -- 172.21.15.7:0/2290222183 learned_addr learned my addr 172.21.15.7:0/2290222183 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:29.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.559+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959154088 0 0) 0x7f98380ab0e0 con 0x7f98380b4670 2024-03-25T18:17:29.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.559+0000 7f9835d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.559+0000 7f9835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.559+0000 7f9835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.559+0000 7f9835d9b640 10 monclient(hunting): my global_id is 4356 2024-03-25T18:17:29.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.559+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9824003520 con 0x7f98380b4670 2024-03-25T18:17:29.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3815177974 0 0) 0x7f98380ab3c0 con 0x7f98380b4ff0 2024-03-25T18:17:29.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): my global_id is 4354 2024-03-25T18:17:29.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98380ab0e0 con 0x7f98380b4ff0 2024-03-25T18:17:29.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073642099 0 0) 0x7f98380a9f30 con 0x7f983813c480 2024-03-25T18:17:29.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 10 monclient(hunting): my global_id is 4376 2024-03-25T18:17:29.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.577+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98380ab3c0 con 0x7f983813c480 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3395306360 0 0) 0x7f9824003520 con 0x7f98380b4670 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 >> v1:172.21.15.7:6790/0 conn(0x7f983813c480 legacy=0x7f983814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 >> v1:172.21.15.183:6789/0 conn(0x7f98380b4ff0 legacy=0x7f983813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 1 monclient: found mon.noname-a 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98380a9b70 con 0x7f98380b4670 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 10 monclient: _finish_auth 0 2024-03-25T18:17:29.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9820001690 con 0x7f98380b4670 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f983cf10640 5 monclient: authenticate success, global_id 4356 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f983cf10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f9835d9b640 10 monclient: dump: 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:29.588 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:29.589 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:29.589 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:29.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.586+0000 7f983cf10640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98200017f0 con 0x7f98380b4670 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f9835d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f9835d9b640 1 -- 172.21.15.7:0/2290222183 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9820001c00 con 0x7f98380b4670 2024-03-25T18:17:29.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f9835d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f9835d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f9835d9b640 10 monclient: dump: 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f983cf10640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:29.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f983cf10640 4 set_mon_vals no callback set 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f983cf10640 10 monclient: shutdown 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f983cf10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.614+0000 7f983cf10640 1 -- 172.21.15.7:0/2290222183 >> v1:172.21.15.7:6789/0 conn(0x7f98380b4670 legacy=0x7f98381425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.615+0000 7f983cf10640 1 -- 172.21.15.7:0/2290222183 shutdown_connections 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.615+0000 7f983cf10640 1 -- 172.21.15.7:0/2290222183 >> 172.21.15.7:0/2290222183 conn(0x7f98380617a0 msgr2=0x7f9838061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.615+0000 7f983cf10640 1 -- 172.21.15.7:0/2290222183 shutdown_connections 2024-03-25T18:17:29.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.615+0000 7f983cf10640 1 -- 172.21.15.7:0/2290222183 wait complete. 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 10 monclient: build_initial_monmap 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 10 monclient: monmap: 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 1 librados: starting msgr at 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 1 librados: starting objecter 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 1 Processor -- start 2024-03-25T18:17:29.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 1 -- start start 2024-03-25T18:17:29.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 1 librados: setting wanted keys 2024-03-25T18:17:29.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 1 librados: calling monclient init 2024-03-25T18:17:29.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.616+0000 7f983cf10640 10 monclient: init 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 10 monclient(hunting): picked mon.noname-c con 0x7f98380b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 10 monclient(hunting): picked mon.noname-b con 0x7f98380b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 10 monclient(hunting): picked mon.noname-a con 0x7f983813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f983813df60 con 0x7f983813c480 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9838144870 con 0x7f98380b4ff0 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9838144ae0 con 0x7f98380b4670 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:29.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.617+0000 7f983cf10640 10 monclient(hunting): authenticate will time out at 7491.721964s 2024-03-25T18:17:29.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.621+0000 7f9836d9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f983813c480 0x7f98381e2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:40250/0 (socket says 172.21.15.7:40250) 2024-03-25T18:17:29.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.621+0000 7f9836d9d640 1 -- 172.21.15.7:0/4091928823 learned_addr learned my addr 172.21.15.7:0/4091928823 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:29.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.626+0000 7f982effd640 10 client.?.objecter ms_handle_connect 0x7f98380b4670 2024-03-25T18:17:29.627 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.626+0000 7f982effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.628+0000 7f982effd640 10 client.?.objecter ms_handle_connect 0x7f983813c480 2024-03-25T18:17:29.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.628+0000 7f982effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.632+0000 7f982effd640 10 client.?.objecter ms_handle_connect 0x7f98380b4ff0 2024-03-25T18:17:29.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.632+0000 7f982effd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:29.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.636+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861979428 0 0) 0x7f9838144870 con 0x7f98380b4ff0 2024-03-25T18:17:29.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.636+0000 7f982effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.636+0000 7f982effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.636+0000 7f982effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.636+0000 7f982effd640 10 monclient(hunting): my global_id is 4357 2024-03-25T18:17:29.637 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.636+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f98380b4ff0 2024-03-25T18:17:29.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.639+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463445643 0 0) 0x7f9838144ae0 con 0x7f98380b4670 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.639+0000 7f982effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.639+0000 7f982effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.639+0000 7f982effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.639+0000 7f982effd640 10 monclient(hunting): my global_id is 4379 2024-03-25T18:17:29.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.639+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9838144870 con 0x7f98380b4670 2024-03-25T18:17:29.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.640+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282534735 0 0) 0x7f983813df60 con 0x7f983813c480 2024-03-25T18:17:29.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.640+0000 7f982effd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:29.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.640+0000 7f982effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:29.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.640+0000 7f982effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:29.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.640+0000 7f982effd640 10 monclient(hunting): my global_id is 4359 2024-03-25T18:17:29.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.640+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9838144ae0 con 0x7f983813c480 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.654+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 614452627 0 0) 0x7f9818004eb0 con 0x7f98380b4ff0 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.654+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9818005620 con 0x7f98380b4ff0 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.654+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9820001b00 con 0x7f98380b4ff0 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.654+0000 7f982effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.654+0000 7f982effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.654+0000 7f982effd640 10 monclient(hunting): dump: 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:29.655 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:29.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.654+0000 7f982effd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.660+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3751589658 0 0) 0x7f9838144870 con 0x7f98380b4670 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.660+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9818004eb0 con 0x7f98380b4670 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.660+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9828001970 con 0x7f98380b4670 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.660+0000 7f982effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.660+0000 7f982effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.660+0000 7f982effd640 10 monclient(hunting): dump: 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:29.661 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:29.662 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.663+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2037486618 0 0) 0x7f9838144ae0 con 0x7f983813c480 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.663+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9838144870 con 0x7f983813c480 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.663+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9830001bd0 con 0x7f983813c480 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.663+0000 7f982effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.663+0000 7f982effd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.663+0000 7f982effd640 10 monclient(hunting): dump: 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:29.664 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:29.665 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:29.665 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:29.665 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:29.665 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:29.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 153667154 0 0) 0x7f9818004eb0 con 0x7f98380b4670 2024-03-25T18:17:29.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 >> v1:172.21.15.183:6789/0 conn(0x7f98380b4ff0 legacy=0x7f9838145a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 >> v1:172.21.15.7:6789/0 conn(0x7f983813c480 legacy=0x7f98381e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:29.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:29.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 1 monclient: found mon.c 2024-03-25T18:17:29.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9838133090 con 0x7f98380b4670 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 10 monclient: _finish_auth 0 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f982effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 5 monclient: authenticate success, global_id 4379 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 10 monclient: _renew_subs 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983813d940 con 0x7f98380b4670 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 10 client.4379.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 10 monclient: _renew_subs 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f983813f540 con 0x7f98380b4670 2024-03-25T18:17:29.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.685+0000 7f983cf10640 1 librados: init done 2024-03-25T18:17:29.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.686+0000 7f983cf10640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.686+0000 7f983cf10640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:29.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.686+0000 7f983cf10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:29.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:29.686+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983813ddd0 con 0x7f98380b4670 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f98280035f0 con 0x7f98380b4670 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f982effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9828003900 con 0x7f98380b4670 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f982effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f982effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f982effd640 10 monclient: dump: 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:30.002 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:30.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f983559a640 4 set_mon_vals no callback set 2024-03-25T18:17:30.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.001+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f98280198d0 con 0x7f98380b4670 2024-03-25T18:17:30.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.002+0000 7f982ffff640 10 monclient: get_auth_request con 0x7f981806e260 auth_method 0 2024-03-25T18:17:30.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.002+0000 7f982effd640 10 client.4379.objecter ms_handle_connect 0x7f981806e260 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f98280535b0 con 0x7f98380b4670 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 10 client.4379.objecter ms_dispatch 0x7f9838000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 3 client.4379.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 3 client.4379.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 20 client.4379.objecter dump_active .. 0 homeless 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f982804eef0 con 0x7f98380b4670 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:30.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.013+0000 7f982effd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:30.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.262+0000 7f983cf10640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:30.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.262+0000 7f983cf10640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:30.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.262+0000 7f983cf10640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:17:30.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.262+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f983806dbd0 con 0x7f98380b4670 2024-03-25T18:17:30.292 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.292+0000 7f982effd640 1 -- 172.21.15.7:0/4091928823 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f9828050ff0 con 0x7f98380b4670 2024-03-25T18:17:30.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.292+0000 7f982effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:30.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.292+0000 7f982effd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:30.293 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:17:30.293 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:30.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 10 librados: watch_flush enter 2024-03-25T18:17:30.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 10 librados: watch_flush exit 2024-03-25T18:17:30.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 20 client.4379.objecter shutdown clearing up homeless session... 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 10 client.4379.objecter successfully canceled tick 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 >> v1:172.21.15.7:6800/1717564767 conn(0x7f981806e260 legacy=0x7f9818070720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 10 monclient: shutdown 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 >> v1:172.21.15.7:6790/0 conn(0x7f98380b4670 legacy=0x7f9838145350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 shutdown_connections 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 >> 172.21.15.7:0/4091928823 conn(0x7f98380617a0 msgr2=0x7f98380b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 shutdown_connections 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 1 -- 172.21.15.7:0/4091928823 wait complete. 2024-03-25T18:17:30.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:30.294+0000 7f983cf10640 1 librados: shutdown 2024-03-25T18:17:30.307 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:37.114 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.230+0000 7fa2f6282640 10 monclient: get_monmap_and_config 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.230+0000 7fa2f6282640 10 monclient: build_initial_monmap 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.230+0000 7fa2f6282640 10 monclient: monmap: 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:17:37.231472+0000 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:17:37.231472+0000 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:17:37.231 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:17:37.232 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.231+0000 7fa2f6282640 1 Processor -- start 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.231+0000 7fa2f6282640 1 -- start start 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.231+0000 7fa2f6282640 10 monclient: init 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f00b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f00b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f00b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f00a96f0 con 0x7fa2f00b4e70 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f00ab0e0 con 0x7fa2f00b1b00 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f0145f70 con 0x7fa2f00b44b0 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:37.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.232+0000 7fa2f6282640 10 monclient(hunting): authenticate will time out at 7499.336818s 2024-03-25T18:17:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.239+0000 7fa2effff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fa2f00b4e70 0x7fa2f013f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49582/0 (socket says 172.21.15.7:49582) 2024-03-25T18:17:37.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.239+0000 7fa2effff640 1 -- 172.21.15.7:0/720995384 learned_addr learned my addr 172.21.15.7:0/720995384 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.254+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1998434595 0 0) 0x7fa2f00a96f0 con 0x7fa2f00b4e70 2024-03-25T18:17:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.254+0000 7fa2eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.254+0000 7fa2eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.254+0000 7fa2eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.254+0000 7fa2eeffd640 10 monclient(hunting): my global_id is 4362 2024-03-25T18:17:37.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.254+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2d8003520 con 0x7fa2f00b4e70 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766294889 0 0) 0x7fa2f0145f70 con 0x7fa2f00b44b0 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 10 monclient(hunting): my global_id is 4382 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f00a96f0 con 0x7fa2f00b44b0 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 59020417 0 0) 0x7fa2d8003520 con 0x7fa2f00b4e70 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 >> v1:172.21.15.7:6790/0 conn(0x7fa2f00b44b0 legacy=0x7fa2f0145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.264+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 >> v1:172.21.15.183:6789/0 conn(0x7fa2f00b1b00 legacy=0x7fa2f014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 1 monclient: found mon.noname-a 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:37.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f00a9b80 con 0x7fa2f00b4e70 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa2e0001690 con 0x7fa2f00b4e70 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2f6282640 5 monclient: authenticate success, global_id 4362 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2f6282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2eeffd640 10 monclient: dump: 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:37.267 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:37.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.265+0000 7fa2f6282640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:37.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.276+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e00017f0 con 0x7fa2f00b4e70 2024-03-25T18:17:37.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.276+0000 7fa2eeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:37.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.276+0000 7fa2eeffd640 1 -- 172.21.15.7:0/720995384 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa2e0001c00 con 0x7fa2f00b4e70 2024-03-25T18:17:37.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.276+0000 7fa2eeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.276+0000 7fa2eeffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.276+0000 7fa2eeffd640 10 monclient: dump: 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:37.278 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:37.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:37.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 4 set_mon_vals no callback set 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 10 monclient: shutdown 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 1 -- 172.21.15.7:0/720995384 >> v1:172.21.15.7:6789/0 conn(0x7fa2f00b4e70 legacy=0x7fa2f013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 1 -- 172.21.15.7:0/720995384 shutdown_connections 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 1 -- 172.21.15.7:0/720995384 >> 172.21.15.7:0/720995384 conn(0x7fa2f00617a0 msgr2=0x7fa2f0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 1 -- 172.21.15.7:0/720995384 shutdown_connections 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.277+0000 7fa2f6282640 1 -- 172.21.15.7:0/720995384 wait complete. 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.278+0000 7fa2f6282640 10 monclient: build_initial_monmap 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.278+0000 7fa2f6282640 10 monclient: monmap: 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:37.280 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.278+0000 7fa2f6282640 1 librados: starting msgr at 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.278+0000 7fa2f6282640 1 librados: starting objecter 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 1 Processor -- start 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 1 -- start start 2024-03-25T18:17:37.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 1 librados: setting wanted keys 2024-03-25T18:17:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 1 librados: calling monclient init 2024-03-25T18:17:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 10 monclient: init 2024-03-25T18:17:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:37.282 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.279+0000 7fa2f6282640 10 monclient(hunting): picked mon.noname-c con 0x7fa2f00b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.280+0000 7fa2f6282640 10 monclient(hunting): picked mon.noname-a con 0x7fa2f00b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.280+0000 7fa2f6282640 10 monclient(hunting): picked mon.noname-b con 0x7fa2f00b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.280+0000 7fa2f6282640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f0141740 con 0x7fa2f00b44b0 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.280+0000 7fa2f6282640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f0144170 con 0x7fa2f00b4e70 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.280+0000 7fa2f6282640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2f0144300 con 0x7fa2f00b1b00 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.280+0000 7fa2f6282640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:37.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.280+0000 7fa2f6282640 10 monclient(hunting): authenticate will time out at 7499.384786s 2024-03-25T18:17:37.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.285+0000 7fa2f481f640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fa2f00b4e70 0x7fa2f0143a60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:59844/0 (socket says 172.21.15.7:59844) 2024-03-25T18:17:37.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.285+0000 7fa2f481f640 1 -- 172.21.15.7:0/3983736011 learned_addr learned my addr 172.21.15.7:0/3983736011 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:37.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.287+0000 7fa2ecff9640 10 client.?.objecter ms_handle_connect 0x7fa2f00b4e70 2024-03-25T18:17:37.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.287+0000 7fa2ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:37.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.292+0000 7fa2ecff9640 10 client.?.objecter ms_handle_connect 0x7fa2f00b1b00 2024-03-25T18:17:37.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.292+0000 7fa2ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:37.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.292+0000 7fa2ecff9640 10 client.?.objecter ms_handle_connect 0x7fa2f00b44b0 2024-03-25T18:17:37.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.292+0000 7fa2ecff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:37.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.296+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209700894 0 0) 0x7fa2f0144170 con 0x7fa2f00b4e70 2024-03-25T18:17:37.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.296+0000 7fa2ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.296+0000 7fa2ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.296+0000 7fa2ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.296+0000 7fa2ecff9640 10 monclient(hunting): my global_id is 4363 2024-03-25T18:17:37.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.296+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2dc004eb0 con 0x7fa2f00b4e70 2024-03-25T18:17:37.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.307+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831607608 0 0) 0x7fa2f0141740 con 0x7fa2f00b44b0 2024-03-25T18:17:37.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.307+0000 7fa2ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.307+0000 7fa2ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.307+0000 7fa2ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.307+0000 7fa2ecff9640 10 monclient(hunting): my global_id is 4365 2024-03-25T18:17:37.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.307+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0144170 con 0x7fa2f00b44b0 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.324+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 593727303 0 0) 0x7fa2dc004eb0 con 0x7fa2f00b4e70 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.324+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2f0141740 con 0x7fa2f00b4e70 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.324+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa2e8001cd0 con 0x7fa2f00b4e70 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.324+0000 7fa2ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.324+0000 7fa2ecff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.324+0000 7fa2ecff9640 10 monclient(hunting): dump: 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:37.325 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:37.326 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:37.326 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:37.326 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:37.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.324+0000 7fa2ecff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.326+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2622815180 0 0) 0x7fa2f0144170 con 0x7fa2f00b44b0 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.326+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa2dc004eb0 con 0x7fa2f00b44b0 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.326+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa2e4001930 con 0x7fa2f00b44b0 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.326+0000 7fa2ecff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.326+0000 7fa2ecff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.326+0000 7fa2ecff9640 10 monclient(hunting): dump: 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.327 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.328 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:37.328 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:37.328 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:37.328 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:37.328 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:37.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.330+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449413048 0 0) 0x7fa2f0144300 con 0x7fa2f00b1b00 2024-03-25T18:17:37.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.330+0000 7fa2ecff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:37.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.330+0000 7fa2ecff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:37.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.330+0000 7fa2ecff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:37.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.330+0000 7fa2ecff9640 10 monclient(hunting): my global_id is 4385 2024-03-25T18:17:37.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.330+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2f0144170 con 0x7fa2f00b1b00 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1775533976 0 0) 0x7fa2dc004eb0 con 0x7fa2f00b44b0 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 >> v1:172.21.15.7:6790/0 conn(0x7fa2f00b1b00 legacy=0x7fa2f01418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 >> v1:172.21.15.183:6789/0 conn(0x7fa2f00b4e70 legacy=0x7fa2f0143a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 1 monclient: found mon.a 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f0144490 con 0x7fa2f00b44b0 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2ecff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 5 monclient: authenticate success, global_id 4365 2024-03-25T18:17:37.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 10 monclient: _renew_subs 2024-03-25T18:17:37.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:37.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f01446e0 con 0x7fa2f00b44b0 2024-03-25T18:17:37.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 10 client.4365.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:37.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 10 monclient: _renew_subs 2024-03-25T18:17:37.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:37.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f01e3420 con 0x7fa2f00b44b0 2024-03-25T18:17:37.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.332+0000 7fa2f6282640 1 librados: init done 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.333+0000 7fa2f6282640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.333+0000 7fa2f6282640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.333+0000 7fa2f6282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:37.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.333+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f004f730 con 0x7fa2f00b44b0 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.667+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa2e4003610 con 0x7fa2f00b44b0 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.667+0000 7fa2ecff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.667+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa2e4003960 con 0x7fa2f00b44b0 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.667+0000 7fa2ecff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.667+0000 7fa2ecff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.667+0000 7fa2ecff9640 10 monclient: dump: 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.668 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.667+0000 7fa2ee7fc640 4 set_mon_vals no callback set 2024-03-25T18:17:37.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.668+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fa2e4003b80 con 0x7fa2f00b44b0 2024-03-25T18:17:37.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.668+0000 7fa2effff640 10 monclient: get_auth_request con 0x7fa2dc06df50 auth_method 0 2024-03-25T18:17:37.670 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.669+0000 7fa2ecff9640 10 client.4365.objecter ms_handle_connect 0x7fa2dc06df50 2024-03-25T18:17:37.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.674+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fa2e4053550 con 0x7fa2f00b44b0 2024-03-25T18:17:37.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.674+0000 7fa2ecff9640 10 client.4365.objecter ms_dispatch 0x7fa2f0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:37.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.674+0000 7fa2ecff9640 3 client.4365.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:37.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.674+0000 7fa2ecff9640 3 client.4365.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:37.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.675+0000 7fa2ecff9640 20 client.4365.objecter dump_active .. 0 homeless 2024-03-25T18:17:37.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.675+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa2e404f060 con 0x7fa2f00b44b0 2024-03-25T18:17:37.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.675+0000 7fa2ecff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:37.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.675+0000 7fa2ecff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:37.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.923+0000 7fa2f6282640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.923+0000 7fa2f6282640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.925 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.923+0000 7fa2f6282640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:37.925 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.923+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa2f00aa090 con 0x7fa2f00b44b0 2024-03-25T18:17:37.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.954+0000 7fa2ecff9640 1 -- 172.21.15.7:0/3983736011 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fa2e4050f90 con 0x7fa2f00b44b0 2024-03-25T18:17:37.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.954+0000 7fa2ecff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:37.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.954+0000 7fa2ecff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:37.956 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:17:37.956 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:37.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 10 librados: watch_flush enter 2024-03-25T18:17:37.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 10 librados: watch_flush exit 2024-03-25T18:17:37.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 20 client.4365.objecter shutdown clearing up homeless session... 2024-03-25T18:17:37.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 10 client.4365.objecter successfully canceled tick 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 >> v1:172.21.15.7:6800/1717564767 conn(0x7fa2dc06df50 legacy=0x7fa2dc070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 10 monclient: shutdown 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 >> v1:172.21.15.7:6789/0 conn(0x7fa2f00b44b0 legacy=0x7fa2f0141030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 shutdown_connections 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 >> 172.21.15.7:0/3983736011 conn(0x7fa2f00617a0 msgr2=0x7fa2f00b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 shutdown_connections 2024-03-25T18:17:37.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 1 -- 172.21.15.7:0/3983736011 wait complete. 2024-03-25T18:17:37.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:37.957+0000 7fa2f6282640 1 librados: shutdown 2024-03-25T18:17:37.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:44.776 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:44.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.888+0000 7f59e623e640 10 monclient: get_monmap_and_config 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.888+0000 7f59e623e640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.888+0000 7f59e623e640 10 monclient: monmap: 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:17:44.889052+0000 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:17:44.889052+0000 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:17:44.889 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.888+0000 7f59e623e640 1 Processor -- start 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.888+0000 7f59e623e640 1 -- start start 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.888+0000 7f59e623e640 10 monclient: init 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 10 monclient(hunting): picked mon.noname-b con 0x7f59e00b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 10 monclient(hunting): picked mon.noname-a con 0x7f59e00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 10 monclient(hunting): picked mon.noname-c con 0x7f59e013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e00ab0e0 con 0x7f59e00b4670 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e00ab3c0 con 0x7f59e013c480 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e00a9f30 con 0x7f59e00b4ff0 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.889+0000 7f59e623e640 10 monclient(hunting): authenticate will time out at 7506.994312s 2024-03-25T18:17:44.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.897+0000 7f59dffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f59e013c480 0x7f59e014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:35222/0 (socket says 172.21.15.7:35222) 2024-03-25T18:17:44.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.897+0000 7f59dffff640 1 -- 172.21.15.7:0/987362542 learned_addr learned my addr 172.21.15.7:0/987362542 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:44.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.903+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3496960160 0 0) 0x7f59e00ab3c0 con 0x7f59e013c480 2024-03-25T18:17:44.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.903+0000 7f59de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.903+0000 7f59de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.903+0000 7f59de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.903+0000 7f59de7fc640 10 monclient(hunting): my global_id is 4366 2024-03-25T18:17:44.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.903+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c8003520 con 0x7f59e013c480 2024-03-25T18:17:44.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.905+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849473603 0 0) 0x7f59e00ab0e0 con 0x7f59e00b4670 2024-03-25T18:17:44.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.905+0000 7f59de7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.905+0000 7f59de7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.905+0000 7f59de7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.905+0000 7f59de7fc640 10 monclient(hunting): my global_id is 4368 2024-03-25T18:17:44.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.905+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e00ab3c0 con 0x7f59e00b4670 2024-03-25T18:17:44.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1410092244 0 0) 0x7f59c8003520 con 0x7f59e013c480 2024-03-25T18:17:44.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 >> v1:172.21.15.7:6790/0 conn(0x7f59e00b4ff0 legacy=0x7f59e013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:17:44.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 >> v1:172.21.15.7:6789/0 conn(0x7f59e00b4670 legacy=0x7f59e01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 1 monclient: found mon.noname-c 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e00a9b70 con 0x7f59e013c480 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f59d8001690 con 0x7f59e013c480 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59de7fc640 10 monclient: dump: 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:44.911 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59e623e640 5 monclient: authenticate success, global_id 4366 2024-03-25T18:17:44.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.909+0000 7f59e623e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:44.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59d80017f0 con 0x7f59e013c480 2024-03-25T18:17:44.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59de7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:44.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59de7fc640 1 -- 172.21.15.7:0/987362542 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f59d8001c00 con 0x7f59e013c480 2024-03-25T18:17:44.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59e623e640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:44.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59e623e640 4 set_mon_vals no callback set 2024-03-25T18:17:44.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59de7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.926 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59de7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.925+0000 7f59de7fc640 10 monclient: dump: 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:44.927 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:44.928 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:44.928 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.926+0000 7f59e623e640 10 monclient: shutdown 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.926+0000 7f59e623e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.926+0000 7f59e623e640 1 -- 172.21.15.7:0/987362542 >> v1:172.21.15.183:6789/0 conn(0x7f59e013c480 legacy=0x7f59e014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.926+0000 7f59e623e640 1 -- 172.21.15.7:0/987362542 shutdown_connections 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.926+0000 7f59e623e640 1 -- 172.21.15.7:0/987362542 >> 172.21.15.7:0/987362542 conn(0x7f59e00617a0 msgr2=0x7f59e0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.926+0000 7f59e623e640 1 -- 172.21.15.7:0/987362542 shutdown_connections 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.926+0000 7f59e623e640 1 -- 172.21.15.7:0/987362542 wait complete. 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.927+0000 7f59e623e640 10 monclient: build_initial_monmap 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.927+0000 7f59e623e640 10 monclient: monmap: 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:44.930 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 1 librados: starting msgr at 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 1 librados: starting objecter 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 1 Processor -- start 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 1 -- start start 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 1 librados: setting wanted keys 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 1 librados: calling monclient init 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 10 monclient: init 2024-03-25T18:17:44.931 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.928+0000 7f59e623e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 10 monclient(hunting): picked mon.noname-a con 0x7f59e00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 10 monclient(hunting): picked mon.noname-b con 0x7f59e00b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 10 monclient(hunting): picked mon.noname-c con 0x7f59e013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e013ebf0 con 0x7f59e00b4670 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e0144440 con 0x7f59e00b4ff0 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59e0144720 con 0x7f59e013c480 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:44.932 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.929+0000 7f59e623e640 10 monclient(hunting): authenticate will time out at 7507.033915s 2024-03-25T18:17:44.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.941+0000 7f59dffff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f59e013c480 0x7f59e01e2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:48562/0 (socket says 172.21.15.7:48562) 2024-03-25T18:17:44.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.941+0000 7f59dffff640 1 -- 172.21.15.7:0/343611095 learned_addr learned my addr 172.21.15.7:0/343611095 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:44.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.943+0000 7f59c7fff640 10 client.?.objecter ms_handle_connect 0x7f59e013c480 2024-03-25T18:17:44.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.943+0000 7f59c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.949+0000 7f59c7fff640 10 client.?.objecter ms_handle_connect 0x7f59e00b4670 2024-03-25T18:17:44.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.949+0000 7f59c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.949+0000 7f59c7fff640 10 client.?.objecter ms_handle_connect 0x7f59e00b4ff0 2024-03-25T18:17:44.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.949+0000 7f59c7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:44.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.958+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520097887 0 0) 0x7f59e013ebf0 con 0x7f59e00b4670 2024-03-25T18:17:44.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.958+0000 7f59c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.958+0000 7f59c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.958+0000 7f59c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.958+0000 7f59c7fff640 10 monclient(hunting): my global_id is 4371 2024-03-25T18:17:44.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.958+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59cc004eb0 con 0x7f59e00b4670 2024-03-25T18:17:44.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.961+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1076207444 0 0) 0x7f59e0144720 con 0x7f59e013c480 2024-03-25T18:17:44.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.961+0000 7f59c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.961+0000 7f59c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.961+0000 7f59c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.961+0000 7f59c7fff640 10 monclient(hunting): my global_id is 4388 2024-03-25T18:17:44.962 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.961+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e013ebf0 con 0x7f59e013c480 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.963+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2152736189 0 0) 0x7f59e0144440 con 0x7f59e00b4ff0 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.963+0000 7f59c7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.963+0000 7f59c7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.963+0000 7f59c7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:44.964 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.963+0000 7f59c7fff640 10 monclient(hunting): my global_id is 4369 2024-03-25T18:17:44.965 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.963+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59e0144720 con 0x7f59e00b4ff0 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.985+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1095142034 0 0) 0x7f59cc004eb0 con 0x7f59e00b4670 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.985+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59e0144440 con 0x7f59e00b4670 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.985+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f59d0001890 con 0x7f59e00b4670 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.985+0000 7f59c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.985+0000 7f59c7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.985+0000 7f59c7fff640 10 monclient(hunting): dump: 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:44.986 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:44.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.985+0000 7f59c7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:44.992 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.991+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2941883346 0 0) 0x7f59e0144720 con 0x7f59e00b4ff0 2024-03-25T18:17:44.992 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.991+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f59cc004eb0 con 0x7f59e00b4ff0 2024-03-25T18:17:44.992 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.991+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f59d4001930 con 0x7f59e00b4ff0 2024-03-25T18:17:44.992 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.991+0000 7f59c7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:44.992 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.991+0000 7f59c7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.991+0000 7f59c7fff640 10 monclient(hunting): dump: 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:44.993 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1257759101 0 0) 0x7f59e0144440 con 0x7f59e00b4670 2024-03-25T18:17:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 >> v1:172.21.15.7:6790/0 conn(0x7f59e013c480 legacy=0x7f59e01e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 >> v1:172.21.15.183:6789/0 conn(0x7f59e00b4ff0 legacy=0x7f59e0145e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:17:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 1 monclient: found mon.a 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59e0144a00 con 0x7f59e00b4670 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 10 monclient: _finish_auth 0 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59c7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 5 monclient: authenticate success, global_id 4371 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 10 monclient: _renew_subs 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:44.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59e0144ce0 con 0x7f59e00b4670 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 10 client.4371.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 10 monclient: _renew_subs 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f59e0143680 con 0x7f59e00b4670 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.994+0000 7f59e623e640 1 librados: init done 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.995+0000 7f59e623e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.995+0000 7f59e623e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.995+0000 7f59e623e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:44.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:44.995+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59e006dbd0 con 0x7f59e00b4670 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f59d0001da0 con 0x7f59e00b4670 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59c7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f59d0003c90 con 0x7f59e00b4670 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59c7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59c7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59c7fff640 10 monclient: dump: 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:45.301 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59ddffb640 4 set_mon_vals no callback set 2024-03-25T18:17:45.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.300+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f59d0019c60 con 0x7f59e00b4670 2024-03-25T18:17:45.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.301+0000 7f59deffd640 10 monclient: get_auth_request con 0x7f59cc06df20 auth_method 0 2024-03-25T18:17:45.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.301+0000 7f59c7fff640 10 client.4371.objecter ms_handle_connect 0x7f59cc06df20 2024-03-25T18:17:45.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.307+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f59d00538e0 con 0x7f59e00b4670 2024-03-25T18:17:45.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.307+0000 7f59c7fff640 10 client.4371.objecter ms_dispatch 0x7f59e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:45.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.307+0000 7f59c7fff640 3 client.4371.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:45.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.307+0000 7f59c7fff640 3 client.4371.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:45.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.308+0000 7f59c7fff640 20 client.4371.objecter dump_active .. 0 homeless 2024-03-25T18:17:45.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.308+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f59d004f3f0 con 0x7f59e00b4670 2024-03-25T18:17:45.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.308+0000 7f59c7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:45.309 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.308+0000 7f59c7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:45.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.566+0000 7f59e623e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:45.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.566+0000 7f59e623e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:45.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.566+0000 7f59e623e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:17:45.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.566+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f59e00688c0 con 0x7f59e00b4670 2024-03-25T18:17:45.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.597+0000 7f59c7fff640 1 -- 172.21.15.7:0/343611095 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f59d0051320 con 0x7f59e00b4670 2024-03-25T18:17:45.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.597+0000 7f59c7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:45.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.597+0000 7f59c7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:45.598 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:17:45.599 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:45.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 10 librados: watch_flush enter 2024-03-25T18:17:45.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 10 librados: watch_flush exit 2024-03-25T18:17:45.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 20 client.4371.objecter shutdown clearing up homeless session... 2024-03-25T18:17:45.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 10 client.4371.objecter successfully canceled tick 2024-03-25T18:17:45.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 >> v1:172.21.15.7:6800/1717564767 conn(0x7f59cc06df20 legacy=0x7f59cc0703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 10 monclient: shutdown 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 >> v1:172.21.15.7:6789/0 conn(0x7f59e00b4670 legacy=0x7f59e0145700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 shutdown_connections 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 >> 172.21.15.7:0/343611095 conn(0x7f59e00617a0 msgr2=0x7f59e0140650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 shutdown_connections 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 1 -- 172.21.15.7:0/343611095 wait complete. 2024-03-25T18:17:45.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:45.600+0000 7f59e623e640 1 librados: shutdown 2024-03-25T18:17:45.615 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:17:48.630 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:55.267 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.638 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:55.276 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.672 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:03:55.309 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.690 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:55.328 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.693 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:55.330 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.717 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:55.343 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.719 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:55.346 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.737 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:03:55.374 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:48.755 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:03:55.382 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:17:52.422 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.520+0000 7ff62e9a5640 10 monclient: get_monmap_and_config 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.520+0000 7ff62e9a5640 10 monclient: build_initial_monmap 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.520+0000 7ff62e9a5640 10 monclient: monmap: 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:17:52.521728+0000 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:17:52.521728+0000 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:52.521 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:52.522 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:52.522 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:17:52.522 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:17:52.522 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.521+0000 7ff62e9a5640 1 Processor -- start 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.521+0000 7ff62e9a5640 1 -- start start 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.521+0000 7ff62e9a5640 10 monclient: init 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 10 monclient(hunting): picked mon.noname-a con 0x7ff6280b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 10 monclient(hunting): picked mon.noname-b con 0x7ff6280b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 10 monclient(hunting): picked mon.noname-c con 0x7ff62813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280ab0e0 con 0x7ff6280b4ff0 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280ab3c0 con 0x7ff62813c480 2024-03-25T18:17:52.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280a9f30 con 0x7ff6280b4670 2024-03-25T18:17:52.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:52.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.522+0000 7ff62e9a5640 10 monclient(hunting): authenticate will time out at 7514.626967s 2024-03-25T18:17:52.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.536+0000 7ff6277fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7ff6280b4670 0x7ff6281425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:34606/0 (socket says 172.21.15.7:34606) 2024-03-25T18:17:52.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.536+0000 7ff6277fe640 1 -- 172.21.15.7:0/2637086320 learned_addr learned my addr 172.21.15.7:0/2637086320 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:52.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.548+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261940581 0 0) 0x7ff6280a9f30 con 0x7ff6280b4670 2024-03-25T18:17:52.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.548+0000 7ff626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.548+0000 7ff626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.548+0000 7ff626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.548+0000 7ff626ffd640 10 monclient(hunting): my global_id is 4391 2024-03-25T18:17:52.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.548+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff610003520 con 0x7ff6280b4670 2024-03-25T18:17:52.558 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.557+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2294251856 0 0) 0x7ff6280ab3c0 con 0x7ff62813c480 2024-03-25T18:17:52.558 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.557+0000 7ff626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.558 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.557+0000 7ff626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.558 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.557+0000 7ff626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.558 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.557+0000 7ff626ffd640 10 monclient(hunting): my global_id is 4372 2024-03-25T18:17:52.558 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.557+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6280a9f30 con 0x7ff62813c480 2024-03-25T18:17:52.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.562+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 837935183 0 0) 0x7ff6280ab0e0 con 0x7ff6280b4ff0 2024-03-25T18:17:52.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.562+0000 7ff626ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.562+0000 7ff626ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.562+0000 7ff626ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.562+0000 7ff626ffd640 10 monclient(hunting): my global_id is 4374 2024-03-25T18:17:52.563 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.562+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6280ab3c0 con 0x7ff6280b4ff0 2024-03-25T18:17:52.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2845095624 0 0) 0x7ff610003520 con 0x7ff6280b4670 2024-03-25T18:17:52.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 >> v1:172.21.15.183:6789/0 conn(0x7ff62813c480 legacy=0x7ff62814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 >> v1:172.21.15.7:6789/0 conn(0x7ff6280b4ff0 legacy=0x7ff62813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 1 monclient: found mon.noname-b 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6280a9b70 con 0x7ff6280b4670 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 10 monclient: _finish_auth 0 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff61c001690 con 0x7ff6280b4670 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff626ffd640 10 monclient: dump: 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.567 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff62e9a5640 5 monclient: authenticate success, global_id 4391 2024-03-25T18:17:52.568 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.565+0000 7ff62e9a5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:17:52.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.577+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff61c0018b0 con 0x7ff6280b4670 2024-03-25T18:17:52.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff626ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:52.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff626ffd640 1 -- 172.21.15.7:0/2637086320 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff61c001c40 con 0x7ff6280b4670 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 10 monclient: get_monmap_and_config success 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 4 set_mon_vals no callback set 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff626ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff626ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff626ffd640 10 monclient: dump: 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:52.579 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 10 monclient: shutdown 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 1 -- 172.21.15.7:0/2637086320 >> v1:172.21.15.7:6790/0 conn(0x7ff6280b4670 legacy=0x7ff6281425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 1 -- 172.21.15.7:0/2637086320 shutdown_connections 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 1 -- 172.21.15.7:0/2637086320 >> 172.21.15.7:0/2637086320 conn(0x7ff6280617a0 msgr2=0x7ff628061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 1 -- 172.21.15.7:0/2637086320 shutdown_connections 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.578+0000 7ff62e9a5640 1 -- 172.21.15.7:0/2637086320 wait complete. 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.579+0000 7ff62e9a5640 10 monclient: build_initial_monmap 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.579+0000 7ff62e9a5640 10 monclient: monmap: 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:17:52.581 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.579+0000 7ff62e9a5640 1 librados: starting msgr at 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.579+0000 7ff62e9a5640 1 librados: starting objecter 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 1 Processor -- start 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 1 -- start start 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 1 librados: setting wanted keys 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 1 librados: calling monclient init 2024-03-25T18:17:52.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 10 monclient: init 2024-03-25T18:17:52.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 10 monclient: _reopen_session rank -1 2024-03-25T18:17:52.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:17:52.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 10 monclient(hunting): picked mon.noname-b con 0x7ff6280b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:17:52.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.580+0000 7ff62e9a5640 10 monclient(hunting): picked mon.noname-c con 0x7ff6280b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:17:52.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.581+0000 7ff62e9a5640 10 monclient(hunting): picked mon.noname-a con 0x7ff62813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:17:52.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.581+0000 7ff62e9a5640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280aee60 con 0x7ff62813c480 2024-03-25T18:17:52.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.581+0000 7ff62e9a5640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280b0ac0 con 0x7ff6280b4670 2024-03-25T18:17:52.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.581+0000 7ff62e9a5640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6280b0da0 con 0x7ff6280b4ff0 2024-03-25T18:17:52.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.581+0000 7ff62e9a5640 10 monclient(hunting): _renew_subs 2024-03-25T18:17:52.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.581+0000 7ff62e9a5640 10 monclient(hunting): authenticate will time out at 7514.685780s 2024-03-25T18:17:52.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.596+0000 7ff627fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff6280b4670 0x7ff6280af2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:51874/0 (socket says 172.21.15.7:51874) 2024-03-25T18:17:52.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.596+0000 7ff627fff640 1 -- 172.21.15.7:0/3131652169 learned_addr learned my addr 172.21.15.7:0/3131652169 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:17:52.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.597+0000 7ff624ff9640 10 client.?.objecter ms_handle_connect 0x7ff6280b4670 2024-03-25T18:17:52.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.597+0000 7ff624ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.603 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.602+0000 7ff624ff9640 10 client.?.objecter ms_handle_connect 0x7ff62813c480 2024-03-25T18:17:52.603 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.602+0000 7ff624ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.605+0000 7ff624ff9640 10 client.?.objecter ms_handle_connect 0x7ff6280b4ff0 2024-03-25T18:17:52.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.605+0000 7ff624ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:17:52.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.618+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987146119 0 0) 0x7ff6280b0ac0 con 0x7ff6280b4670 2024-03-25T18:17:52.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.618+0000 7ff624ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.618+0000 7ff624ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.618+0000 7ff624ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.618+0000 7ff624ff9640 10 monclient(hunting): my global_id is 4375 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.618+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff614004eb0 con 0x7ff6280b4670 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.619+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1653009414 0 0) 0x7ff6280aee60 con 0x7ff62813c480 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.619+0000 7ff624ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.619+0000 7ff624ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.619+0000 7ff624ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.619+0000 7ff624ff9640 10 monclient(hunting): my global_id is 4377 2024-03-25T18:17:52.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.619+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6280b0ac0 con 0x7ff62813c480 2024-03-25T18:17:52.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.620+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 224004399 0 0) 0x7ff6280b0da0 con 0x7ff6280b4ff0 2024-03-25T18:17:52.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.620+0000 7ff624ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:17:52.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.620+0000 7ff624ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:17:52.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.620+0000 7ff624ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:17:52.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.620+0000 7ff624ff9640 10 monclient(hunting): my global_id is 4394 2024-03-25T18:17:52.621 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.620+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6280aee60 con 0x7ff6280b4ff0 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.640+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024334385 0 0) 0x7ff614004eb0 con 0x7ff6280b4670 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.640+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6280b0da0 con 0x7ff6280b4670 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.640+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff618001a10 con 0x7ff6280b4670 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.640+0000 7ff624ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.640+0000 7ff624ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.640+0000 7ff624ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:52.641 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:52.642 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:52.642 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.640+0000 7ff624ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.661+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3830817996 0 0) 0x7ff6280aee60 con 0x7ff6280b4ff0 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.661+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff614004eb0 con 0x7ff6280b4ff0 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.661+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff61c0015d0 con 0x7ff6280b4ff0 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.661+0000 7ff624ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.661+0000 7ff624ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.661+0000 7ff624ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:52.662 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:52.663 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:52.663 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:52.663 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.663+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1721827034 0 0) 0x7ff6280b0ac0 con 0x7ff62813c480 2024-03-25T18:17:52.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.663+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6280aee60 con 0x7ff62813c480 2024-03-25T18:17:52.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.663+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff620001b50 con 0x7ff62813c480 2024-03-25T18:17:52.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.663+0000 7ff624ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.663+0000 7ff624ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:17:52.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.663+0000 7ff624ff9640 10 monclient(hunting): dump: 2024-03-25T18:17:52.664 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:52.665 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.665+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 36123238 0 0) 0x7ff6280b0da0 con 0x7ff6280b4670 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.665+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 >> v1:172.21.15.7:6790/0 conn(0x7ff6280b4ff0 legacy=0x7ff62813a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 >> v1:172.21.15.7:6789/0 conn(0x7ff62813c480 legacy=0x7ff62813ab60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 1 monclient: found mon.b 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff62813b270 con 0x7ff6280b4670 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 10 monclient: _finish_auth 0 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff624ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:17:52.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 5 monclient: authenticate success, global_id 4375 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 10 monclient: _renew_subs 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6280abd60 con 0x7ff6280b4670 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 10 client.4375.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 10 monclient: _renew_subs 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6280afbe0 con 0x7ff6280b4670 2024-03-25T18:17:52.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.666+0000 7ff62e9a5640 1 librados: init done 2024-03-25T18:17:52.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.667+0000 7ff62e9a5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.667+0000 7ff62e9a5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.667+0000 7ff62e9a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:17:52.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.667+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6280b3ff0 con 0x7ff6280b4670 2024-03-25T18:17:52.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.940+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff618001c70 con 0x7ff6280b4670 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.940+0000 7ff624ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.940+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff618003a60 con 0x7ff6280b4670 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.940+0000 7ff624ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.940+0000 7ff624ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.941+0000 7ff624ff9640 10 monclient: dump: 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:17:52.942 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:17:52.943 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:17:52.943 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:17:52.943 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:17:52.943 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:17:52.943 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:17:52.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.941+0000 7ff6267fc640 4 set_mon_vals no callback set 2024-03-25T18:17:52.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.942+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7ff618003c80 con 0x7ff6280b4670 2024-03-25T18:17:52.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.942+0000 7ff6277fe640 10 monclient: get_auth_request con 0x7ff61406e0f0 auth_method 0 2024-03-25T18:17:52.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.942+0000 7ff624ff9640 10 client.4375.objecter ms_handle_connect 0x7ff61406e0f0 2024-03-25T18:17:52.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.946+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7ff6180548b0 con 0x7ff6280b4670 2024-03-25T18:17:52.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.946+0000 7ff624ff9640 10 client.4375.objecter ms_dispatch 0x7ff628000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:17:52.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.946+0000 7ff624ff9640 3 client.4375.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:17:52.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.946+0000 7ff624ff9640 3 client.4375.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:17:52.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.946+0000 7ff624ff9640 20 client.4375.objecter dump_active .. 0 homeless 2024-03-25T18:17:52.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.946+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6180503c0 con 0x7ff6280b4670 2024-03-25T18:17:52.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.947+0000 7ff624ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:17:52.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:52.947+0000 7ff624ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:17:53.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.197+0000 7ff62e9a5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:53.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.197+0000 7ff62e9a5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:53.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.197+0000 7ff62e9a5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:17:53.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.197+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff62806dbd0 con 0x7ff6280b4670 2024-03-25T18:17:53.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.235+0000 7ff624ff9640 1 -- 172.21.15.7:0/3131652169 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7ff6180522f0 con 0x7ff6280b4670 2024-03-25T18:17:53.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.235+0000 7ff624ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:17:53.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.235+0000 7ff624ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:17:53.236 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:17:53.236 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 10 librados: watch_flush enter 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 10 librados: watch_flush exit 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 20 client.4375.objecter shutdown clearing up homeless session... 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 10 client.4375.objecter successfully canceled tick 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 >> v1:172.21.15.7:6800/1717564767 conn(0x7ff61406e0f0 legacy=0x7ff6140705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 10 monclient: shutdown 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 >> v1:172.21.15.183:6789/0 conn(0x7ff6280b4670 legacy=0x7ff6280af2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:17:53.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 shutdown_connections 2024-03-25T18:17:53.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.237+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 >> 172.21.15.7:0/3131652169 conn(0x7ff6280617a0 msgr2=0x7ff6280b6cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:17:53.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.238+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 shutdown_connections 2024-03-25T18:17:53.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.238+0000 7ff62e9a5640 1 -- 172.21.15.7:0/3131652169 wait complete. 2024-03-25T18:17:53.240 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:17:53.238+0000 7ff62e9a5640 1 librados: shutdown 2024-03-25T18:17:53.250 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:00.057 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:00.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.171+0000 7ffad7b93640 10 monclient: get_monmap_and_config 2024-03-25T18:18:00.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.171+0000 7ffad7b93640 10 monclient: build_initial_monmap 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.171+0000 7ffad7b93640 10 monclient: monmap: 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:00.172822+0000 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:00.172822+0000 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:00.173 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:00.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.172+0000 7ffad7b93640 1 Processor -- start 2024-03-25T18:18:00.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.172+0000 7ffad7b93640 1 -- start start 2024-03-25T18:18:00.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.172+0000 7ffad7b93640 10 monclient: init 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 10 monclient(hunting): picked mon.noname-b con 0x7ffad00b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 10 monclient(hunting): picked mon.noname-c con 0x7ffad00b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 10 monclient(hunting): picked mon.noname-a con 0x7ffad013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00ab0e0 con 0x7ffad013c480 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00ab3c0 con 0x7ffad00b4670 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad00a9f30 con 0x7ffad00b4ff0 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:00.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.173+0000 7ffad7b93640 10 monclient(hunting): authenticate will time out at 7522.278144s 2024-03-25T18:18:00.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.181+0000 7ffad592f640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7ffad00b4ff0 0x7ffad013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:39412/0 (socket says 172.21.15.7:39412) 2024-03-25T18:18:00.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.181+0000 7ffad592f640 1 -- 172.21.15.7:0/1469249902 learned_addr learned my addr 172.21.15.7:0/1469249902 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:00.187 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.186+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3946086549 0 0) 0x7ffad00a9f30 con 0x7ffad00b4ff0 2024-03-25T18:18:00.187 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.186+0000 7ffad492d640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.187 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.186+0000 7ffad492d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.187 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.186+0000 7ffad492d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.187 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.186+0000 7ffad492d640 10 monclient(hunting): my global_id is 4397 2024-03-25T18:18:00.187 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.186+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffabc003520 con 0x7ffad00b4ff0 2024-03-25T18:18:00.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.194+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2273541718 0 0) 0x7ffad00ab3c0 con 0x7ffad00b4670 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): my global_id is 4378 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad00a9f30 con 0x7ffad00b4670 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 920554753 0 0) 0x7ffad00ab0e0 con 0x7ffad013c480 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 10 monclient(hunting): my global_id is 4380 2024-03-25T18:18:00.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.195+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad00ab3c0 con 0x7ffad013c480 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1067935746 0 0) 0x7ffabc003520 con 0x7ffad00b4ff0 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 >> v1:172.21.15.183:6789/0 conn(0x7ffad00b4670 legacy=0x7ffad01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 >> v1:172.21.15.7:6789/0 conn(0x7ffad013c480 legacy=0x7ffad014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 10 monclient: _finish_hunting 0 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 1 monclient: found mon.noname-b 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad00a9b70 con 0x7ffad00b4ff0 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 10 monclient: _finish_auth 0 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 10 monclient: _check_auth_tickets 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ffacc001690 con 0x7ffad00b4ff0 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:00.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad492d640 10 monclient: dump: 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad7b93640 5 monclient: authenticate success, global_id 4397 2024-03-25T18:18:00.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.202+0000 7ffad7b93640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:00.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.208+0000 7ffad492d640 1 -- 172.21.15.7:0/1469249902 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffacc0017f0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.208+0000 7ffad492d640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:00.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.208+0000 7ffad7b93640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:00.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.208+0000 7ffad7b93640 4 set_mon_vals no callback set 2024-03-25T18:18:00.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.209+0000 7ffad7b93640 10 monclient: shutdown 2024-03-25T18:18:00.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.209+0000 7ffad7b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:00.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.209+0000 7ffad7b93640 1 -- 172.21.15.7:0/1469249902 >> v1:172.21.15.7:6790/0 conn(0x7ffad00b4ff0 legacy=0x7ffad013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.209+0000 7ffad7b93640 1 -- 172.21.15.7:0/1469249902 shutdown_connections 2024-03-25T18:18:00.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.209+0000 7ffad7b93640 1 -- 172.21.15.7:0/1469249902 >> 172.21.15.7:0/1469249902 conn(0x7ffad00617a0 msgr2=0x7ffad0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:00.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.209+0000 7ffad7b93640 1 -- 172.21.15.7:0/1469249902 shutdown_connections 2024-03-25T18:18:00.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.209+0000 7ffad7b93640 1 -- 172.21.15.7:0/1469249902 wait complete. 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.210+0000 7ffad7b93640 10 monclient: build_initial_monmap 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.210+0000 7ffad7b93640 10 monclient: monmap: 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:00.211 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:00.212 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:00.212 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:00.212 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:00.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.211+0000 7ffad7b93640 1 librados: starting msgr at 2024-03-25T18:18:00.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.211+0000 7ffad7b93640 1 librados: starting objecter 2024-03-25T18:18:00.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.211+0000 7ffad7b93640 1 Processor -- start 2024-03-25T18:18:00.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.211+0000 7ffad7b93640 1 -- start start 2024-03-25T18:18:00.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.211+0000 7ffad7b93640 1 librados: setting wanted keys 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.211+0000 7ffad7b93640 1 librados: calling monclient init 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.211+0000 7ffad7b93640 10 monclient: init 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 10 monclient(hunting): picked mon.noname-a con 0x7ffad00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 10 monclient(hunting): picked mon.noname-b con 0x7ffad00b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 10 monclient(hunting): picked mon.noname-c con 0x7ffad013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad013f8f0 con 0x7ffad00b4670 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad013e0e0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffad013f510 con 0x7ffad013c480 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:00.214 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.212+0000 7ffad7b93640 10 monclient(hunting): authenticate will time out at 7522.317028s 2024-03-25T18:18:00.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.227+0000 7ffad512e640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ffad00b4ff0 0x7ffad01dbd30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:56680/0 (socket says 172.21.15.7:56680) 2024-03-25T18:18:00.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.227+0000 7ffad512e640 1 -- 172.21.15.7:0/2371115562 learned_addr learned my addr 172.21.15.7:0/2371115562 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:00.230 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.229+0000 7ffac67fc640 10 client.?.objecter ms_handle_connect 0x7ffad00b4ff0 2024-03-25T18:18:00.230 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.229+0000 7ffac67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:00.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.235+0000 7ffac67fc640 10 client.?.objecter ms_handle_connect 0x7ffad00b4670 2024-03-25T18:18:00.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.235+0000 7ffac67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:00.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.237+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3650335742 0 0) 0x7ffad013e0e0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.237+0000 7ffac67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.237+0000 7ffac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.237+0000 7ffac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.237+0000 7ffac67fc640 10 monclient(hunting): my global_id is 4381 2024-03-25T18:18:00.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.237+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffab4004eb0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.242+0000 7ffac67fc640 10 client.?.objecter ms_handle_connect 0x7ffad013c480 2024-03-25T18:18:00.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.242+0000 7ffac67fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:00.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.248+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2128562436 0 0) 0x7ffad013f8f0 con 0x7ffad00b4670 2024-03-25T18:18:00.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.248+0000 7ffac67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.248+0000 7ffac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.248+0000 7ffac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.248+0000 7ffac67fc640 10 monclient(hunting): my global_id is 4383 2024-03-25T18:18:00.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.248+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad013e0e0 con 0x7ffad00b4670 2024-03-25T18:18:00.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.251+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3181022947 0 0) 0x7ffad013f510 con 0x7ffad013c480 2024-03-25T18:18:00.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.251+0000 7ffac67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:00.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.251+0000 7ffac67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:00.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.251+0000 7ffac67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:00.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.251+0000 7ffac67fc640 10 monclient(hunting): my global_id is 4400 2024-03-25T18:18:00.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.251+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffad013f8f0 con 0x7ffad013c480 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.255+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2779379431 0 0) 0x7ffab4004eb0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.255+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ffad013f510 con 0x7ffad00b4ff0 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.255+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ffac0001a30 con 0x7ffad00b4ff0 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.255+0000 7ffac67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.255+0000 7ffac67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.255+0000 7ffac67fc640 10 monclient(hunting): dump: 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:00.256 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:00.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.255+0000 7ffac67fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 880197517 0 0) 0x7ffad013f510 con 0x7ffad00b4ff0 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 >> v1:172.21.15.7:6790/0 conn(0x7ffad013c480 legacy=0x7ffad01e28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 >> v1:172.21.15.7:6789/0 conn(0x7ffad00b4670 legacy=0x7ffad01db620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 1 monclient: found mon.b 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffad0145af0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffac67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffad7b93640 5 monclient: authenticate success, global_id 4381 2024-03-25T18:18:00.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffad7b93640 10 monclient: _renew_subs 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffad7b93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.263+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffad0146590 con 0x7ffad00b4ff0 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.264+0000 7ffad7b93640 10 client.4381.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.264+0000 7ffad7b93640 10 monclient: _renew_subs 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.264+0000 7ffad7b93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.264+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ffad013ccc0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.264+0000 7ffad7b93640 1 librados: init done 2024-03-25T18:18:00.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.265+0000 7ffad7b93640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:00.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.265+0000 7ffad7b93640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:00.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.265+0000 7ffad7b93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:00.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.265+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffad013f6a0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.541+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ffac0001c70 con 0x7ffad00b4ff0 2024-03-25T18:18:00.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.541+0000 7ffac67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:00.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.541+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ffac0003a60 con 0x7ffad00b4ff0 2024-03-25T18:18:00.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.541+0000 7ffac67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.541+0000 7ffac67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.541+0000 7ffac67fc640 10 monclient: dump: 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:00.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.541+0000 7ffac7fff640 4 set_mon_vals no callback set 2024-03-25T18:18:00.544 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.542+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7ffac0003c80 con 0x7ffad00b4ff0 2024-03-25T18:18:00.544 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.543+0000 7ffad592f640 10 monclient: get_auth_request con 0x7ffab406ddf0 auth_method 0 2024-03-25T18:18:00.544 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.543+0000 7ffac67fc640 10 client.4381.objecter ms_handle_connect 0x7ffab406ddf0 2024-03-25T18:18:00.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.548+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7ffac00548f0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.548+0000 7ffac67fc640 10 client.4381.objecter ms_dispatch 0x7ffad0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:00.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.548+0000 7ffac67fc640 3 client.4381.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:00.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.548+0000 7ffac67fc640 3 client.4381.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:00.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.548+0000 7ffac67fc640 20 client.4381.objecter dump_active .. 0 homeless 2024-03-25T18:18:00.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.550+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ffac0050400 con 0x7ffad00b4ff0 2024-03-25T18:18:00.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.550+0000 7ffac67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:00.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.550+0000 7ffac67fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:00.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.818+0000 7ffad7b93640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.818+0000 7ffad7b93640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.818+0000 7ffad7b93640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:00.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.818+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ffad006dbd0 con 0x7ffad00b4ff0 2024-03-25T18:18:00.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.860+0000 7ffac67fc640 1 -- 172.21.15.7:0/2371115562 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7ffac0052330 con 0x7ffad00b4ff0 2024-03-25T18:18:00.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.860+0000 7ffac67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:00.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.860+0000 7ffac67fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:00.861 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:00.861 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 10 librados: watch_flush enter 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 10 librados: watch_flush exit 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 20 client.4381.objecter shutdown clearing up homeless session... 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 10 client.4381.objecter successfully canceled tick 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 >> v1:172.21.15.7:6800/1717564767 conn(0x7ffab406ddf0 legacy=0x7ffab40702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 10 monclient: shutdown 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:00.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 >> v1:172.21.15.183:6789/0 conn(0x7ffad00b4ff0 legacy=0x7ffad01dbd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:00.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 shutdown_connections 2024-03-25T18:18:00.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 >> 172.21.15.7:0/2371115562 conn(0x7ffad00617a0 msgr2=0x7ffad0140780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:00.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 shutdown_connections 2024-03-25T18:18:00.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 1 -- 172.21.15.7:0/2371115562 wait complete. 2024-03-25T18:18:00.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:00.862+0000 7ffad7b93640 1 librados: shutdown 2024-03-25T18:18:00.876 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:07.683 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.801+0000 7f27bcd2e640 10 monclient: get_monmap_and_config 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.801+0000 7f27bcd2e640 10 monclient: build_initial_monmap 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.801+0000 7f27bcd2e640 10 monclient: monmap: 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:07.802445+0000 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:07.802445+0000 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:07.802 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:07.803 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:07.803 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:07.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 1 Processor -- start 2024-03-25T18:18:07.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 1 -- start start 2024-03-25T18:18:07.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient: init 2024-03-25T18:18:07.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient(hunting): picked mon.noname-b con 0x7f27b80b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient(hunting): picked mon.noname-c con 0x7f27b80b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient(hunting): picked mon.noname-a con 0x7f27b813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b80ab0e0 con 0x7f27b813c480 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b80ab3c0 con 0x7f27b80b4670 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b80a9f30 con 0x7f27b80b4ff0 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:07.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.802+0000 7f27bcd2e640 10 monclient(hunting): authenticate will time out at 7529.907486s 2024-03-25T18:18:07.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.812+0000 7f27b5d9b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f27b80b4670 0x7f27b81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:53672/0 (socket says 172.21.15.7:53672) 2024-03-25T18:18:07.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.812+0000 7f27b5d9b640 1 -- 172.21.15.7:0/250674158 learned_addr learned my addr 172.21.15.7:0/250674158 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:07.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.817+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1832797124 0 0) 0x7f27b80ab3c0 con 0x7f27b80b4670 2024-03-25T18:18:07.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.817+0000 7f27b559a640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.817+0000 7f27b559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.817+0000 7f27b559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.817+0000 7f27b559a640 10 monclient(hunting): my global_id is 4384 2024-03-25T18:18:07.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.817+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27a4003520 con 0x7f27b80b4670 2024-03-25T18:18:07.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3297507668 0 0) 0x7f27a4003520 con 0x7f27b80b4670 2024-03-25T18:18:07.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 >> v1:172.21.15.7:6790/0 conn(0x7f27b80b4ff0 legacy=0x7f27b813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 >> v1:172.21.15.7:6789/0 conn(0x7f27b813c480 legacy=0x7f27b814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 10 monclient: _finish_hunting 0 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 1 monclient: found mon.noname-c 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b80a9b70 con 0x7f27b80b4670 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 10 monclient: _finish_auth 0 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 10 monclient: _check_auth_tickets 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27a0001690 con 0x7f27b80b4670 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27b559a640 10 monclient: dump: 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:07.832 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27bcd2e640 5 monclient: authenticate success, global_id 4384 2024-03-25T18:18:07.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.830+0000 7f27bcd2e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:07.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a00017f0 con 0x7f27b80b4670 2024-03-25T18:18:07.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27b559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:07.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27bcd2e640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:07.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27bcd2e640 4 set_mon_vals no callback set 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27bcd2e640 10 monclient: shutdown 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27bcd2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27bcd2e640 1 -- 172.21.15.7:0/250674158 >> v1:172.21.15.183:6789/0 conn(0x7f27b80b4670 legacy=0x7f27b81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27b559a640 1 -- 172.21.15.7:0/250674158 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27a0001c00 con 0x7f27b80b4670 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.844+0000 7f27b559a640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.845+0000 7f27bcd2e640 1 -- 172.21.15.7:0/250674158 shutdown_connections 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.845+0000 7f27bcd2e640 1 -- 172.21.15.7:0/250674158 >> 172.21.15.7:0/250674158 conn(0x7f27b80617a0 msgr2=0x7f27b8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.845+0000 7f27bcd2e640 1 -- 172.21.15.7:0/250674158 shutdown_connections 2024-03-25T18:18:07.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.845+0000 7f27bcd2e640 1 -- 172.21.15.7:0/250674158 wait complete. 2024-03-25T18:18:07.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 10 monclient: build_initial_monmap 2024-03-25T18:18:07.847 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 10 monclient: monmap: 2024-03-25T18:18:07.847 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:07.847 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 1 librados: starting msgr at 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 1 librados: starting objecter 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 1 Processor -- start 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 1 -- start start 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 1 librados: setting wanted keys 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 1 librados: calling monclient init 2024-03-25T18:18:07.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.846+0000 7f27bcd2e640 10 monclient: init 2024-03-25T18:18:07.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:07.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:07.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 10 monclient(hunting): picked mon.noname-b con 0x7f27b80b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:07.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 10 monclient(hunting): picked mon.noname-c con 0x7f27b80b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:07.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 10 monclient(hunting): picked mon.noname-a con 0x7f27b813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:07.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b813dfc0 con 0x7f27b813c480 2024-03-25T18:18:07.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b813fa00 con 0x7f27b80b4670 2024-03-25T18:18:07.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27b813f740 con 0x7f27b80b4ff0 2024-03-25T18:18:07.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:07.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.847+0000 7f27bcd2e640 10 monclient(hunting): authenticate will time out at 7529.952264s 2024-03-25T18:18:07.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.855+0000 7f27b659c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f27b80b4670 0x7f27b81db480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:53712/0 (socket says 172.21.15.7:53712) 2024-03-25T18:18:07.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.855+0000 7f27b659c640 1 -- 172.21.15.7:0/2455716392 learned_addr learned my addr 172.21.15.7:0/2455716392 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:07.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.856+0000 7f27aeffd640 10 client.?.objecter ms_handle_connect 0x7f27b80b4670 2024-03-25T18:18:07.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.856+0000 7f27aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.857+0000 7f27aeffd640 10 client.?.objecter ms_handle_connect 0x7f27b80b4ff0 2024-03-25T18:18:07.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.857+0000 7f27aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.863+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2536976652 0 0) 0x7f27b813fa00 con 0x7f27b80b4670 2024-03-25T18:18:07.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.863+0000 7f27aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.863+0000 7f27aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.863+0000 7f27aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.863+0000 7f27aeffd640 10 monclient(hunting): my global_id is 4387 2024-03-25T18:18:07.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.863+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2798004eb0 con 0x7f27b80b4670 2024-03-25T18:18:07.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.868+0000 7f27aeffd640 10 client.?.objecter ms_handle_connect 0x7f27b813c480 2024-03-25T18:18:07.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.868+0000 7f27aeffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:07.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.871+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3723298360 0 0) 0x7f2798004eb0 con 0x7f27b80b4670 2024-03-25T18:18:07.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.871+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27b813fa00 con 0x7f27b80b4670 2024-03-25T18:18:07.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.877+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27a8001e90 con 0x7f27b80b4670 2024-03-25T18:18:07.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.877+0000 7f27aeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:07.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.877+0000 7f27aeffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:07.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.877+0000 7f27aeffd640 10 monclient(hunting): dump: 2024-03-25T18:18:07.878 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:07.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.878+0000 7f27aeffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:07.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.880+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1645975508 0 0) 0x7f27b813f740 con 0x7f27b80b4ff0 2024-03-25T18:18:07.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.880+0000 7f27aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.880+0000 7f27aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.880+0000 7f27aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.880+0000 7f27aeffd640 10 monclient(hunting): my global_id is 4406 2024-03-25T18:18:07.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.880+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2798004eb0 con 0x7f27b80b4ff0 2024-03-25T18:18:07.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728733983 0 0) 0x7f27b813dfc0 con 0x7f27b813c480 2024-03-25T18:18:07.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:07.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:07.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:07.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient(hunting): my global_id is 4389 2024-03-25T18:18:07.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b813f740 con 0x7f27b813c480 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2409912369 0 0) 0x7f27b813fa00 con 0x7f27b80b4670 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 >> v1:172.21.15.7:6790/0 conn(0x7f27b80b4ff0 legacy=0x7f27b81dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 >> v1:172.21.15.7:6789/0 conn(0x7f27b813c480 legacy=0x7f27b81e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 1 monclient: found mon.b 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27b8144f80 con 0x7f27b80b4670 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27aeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 5 monclient: authenticate success, global_id 4387 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 10 monclient: _renew_subs 2024-03-25T18:18:07.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:07.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27b8145ed0 con 0x7f27b80b4670 2024-03-25T18:18:07.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 10 client.4387.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:07.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 10 monclient: _renew_subs 2024-03-25T18:18:07.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:07.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27b8143840 con 0x7f27b80b4670 2024-03-25T18:18:07.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.886+0000 7f27bcd2e640 1 librados: init done 2024-03-25T18:18:07.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.887+0000 7f27bcd2e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.887+0000 7f27bcd2e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:07.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.887+0000 7f27bcd2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:07.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:07.887+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27b806dbd0 con 0x7f27b80b4670 2024-03-25T18:18:08.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.159+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27a8001ff0 con 0x7f27b80b4670 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.159+0000 7f27aeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.159+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27a8003de0 con 0x7f27b80b4670 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.159+0000 7f27aeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.159+0000 7f27aeffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.159+0000 7f27aeffd640 10 monclient: dump: 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:08.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.159+0000 7f27b4d99640 4 set_mon_vals no callback set 2024-03-25T18:18:08.162 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.160+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f27a8004000 con 0x7f27b80b4670 2024-03-25T18:18:08.162 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.161+0000 7f27b5d9b640 10 monclient: get_auth_request con 0x7f279806de00 auth_method 0 2024-03-25T18:18:08.162 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.161+0000 7f27aeffd640 10 client.4387.objecter ms_handle_connect 0x7f279806de00 2024-03-25T18:18:08.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f27a8054ce0 con 0x7f27b80b4670 2024-03-25T18:18:08.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 10 client.4387.objecter ms_dispatch 0x7f27b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:08.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 3 client.4387.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:08.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 3 client.4387.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:08.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 20 client.4387.objecter dump_active .. 0 homeless 2024-03-25T18:18:08.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27a80507f0 con 0x7f27b80b4670 2024-03-25T18:18:08.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:08.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.167+0000 7f27aeffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:08.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.419+0000 7f27bcd2e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:08.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.419+0000 7f27bcd2e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:08.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.419+0000 7f27bcd2e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:08.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.419+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27b8143410 con 0x7f27b80b4670 2024-03-25T18:18:08.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.442+0000 7f27aeffd640 1 -- 172.21.15.7:0/2455716392 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f27a8052720 con 0x7f27b80b4670 2024-03-25T18:18:08.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.442+0000 7f27aeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:08.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.442+0000 7f27aeffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:08.443 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:08.443 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:08.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.444+0000 7f27bcd2e640 10 librados: watch_flush enter 2024-03-25T18:18:08.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.444+0000 7f27bcd2e640 10 librados: watch_flush exit 2024-03-25T18:18:08.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.444+0000 7f27bcd2e640 20 client.4387.objecter shutdown clearing up homeless session... 2024-03-25T18:18:08.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.444+0000 7f27bcd2e640 10 client.4387.objecter successfully canceled tick 2024-03-25T18:18:08.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 >> v1:172.21.15.7:6800/1717564767 conn(0x7f279806de00 legacy=0x7f27980702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 10 monclient: shutdown 2024-03-25T18:18:08.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:08.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 >> v1:172.21.15.183:6789/0 conn(0x7f27b80b4670 legacy=0x7f27b81db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:08.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 shutdown_connections 2024-03-25T18:18:08.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 >> 172.21.15.7:0/2455716392 conn(0x7f27b80617a0 msgr2=0x7f27b81407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:08.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 shutdown_connections 2024-03-25T18:18:08.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 1 -- 172.21.15.7:0/2455716392 wait complete. 2024-03-25T18:18:08.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:08.445+0000 7f27bcd2e640 1 librados: shutdown 2024-03-25T18:18:08.457 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:15.264 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:15.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.379+0000 7fb3d4bf6640 10 monclient: get_monmap_and_config 2024-03-25T18:18:15.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.379+0000 7fb3d4bf6640 10 monclient: build_initial_monmap 2024-03-25T18:18:15.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.379+0000 7fb3d4bf6640 10 monclient: monmap: 2024-03-25T18:18:15.380 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:15.380708+0000 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:15.380708+0000 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:15.381 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.380+0000 7fb3d4bf6640 1 Processor -- start 2024-03-25T18:18:15.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.380+0000 7fb3d4bf6640 1 -- start start 2024-03-25T18:18:15.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.380+0000 7fb3d4bf6640 10 monclient: init 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3d00b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3d013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00ab0e0 con 0x7fb3d00b4670 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00ab3c0 con 0x7fb3d013c480 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00a9f30 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:15.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.381+0000 7fb3d4bf6640 10 monclient(hunting): authenticate will time out at 7537.485993s 2024-03-25T18:18:15.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.391+0000 7fb3ced9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb3d013c480 0x7fb3d014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:58840/0 (socket says 172.21.15.7:58840) 2024-03-25T18:18:15.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.391+0000 7fb3ced9d640 1 -- 172.21.15.7:0/3815323126 learned_addr learned my addr 172.21.15.7:0/3815323126 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:15.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.406+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1759619947 0 0) 0x7fb3d00a9f30 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.406+0000 7fb3cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.406+0000 7fb3cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.406+0000 7fb3cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.406+0000 7fb3cdd9b640 10 monclient(hunting): my global_id is 4409 2024-03-25T18:18:15.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.406+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3bc003520 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.407+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 327688051 0 0) 0x7fb3d00ab3c0 con 0x7fb3d013c480 2024-03-25T18:18:15.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.407+0000 7fb3cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.407+0000 7fb3cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.407+0000 7fb3cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.407+0000 7fb3cdd9b640 10 monclient(hunting): my global_id is 4390 2024-03-25T18:18:15.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.407+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d00a9f30 con 0x7fb3d013c480 2024-03-25T18:18:15.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.420+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4275880241 0 0) 0x7fb3d00ab0e0 con 0x7fb3d00b4670 2024-03-25T18:18:15.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.421+0000 7fb3cdd9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.421+0000 7fb3cdd9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.421+0000 7fb3cdd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.421+0000 7fb3cdd9b640 10 monclient(hunting): my global_id is 4392 2024-03-25T18:18:15.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.421+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d00ab3c0 con 0x7fb3d00b4670 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1094435984 0 0) 0x7fb3bc003520 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 >> v1:172.21.15.183:6789/0 conn(0x7fb3d013c480 legacy=0x7fb3d014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 >> v1:172.21.15.7:6789/0 conn(0x7fb3d00b4670 legacy=0x7fb3d01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 1 monclient: found mon.noname-b 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d00a9b70 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb3c0001690 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3d4bf6640 5 monclient: authenticate success, global_id 4409 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3d4bf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3cdd9b640 10 monclient: dump: 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:15.427 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:15.428 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:15.428 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:15.428 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.425+0000 7fb3d4bf6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:15.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3c00017f0 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3cdd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:15.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3cdd9b640 1 -- 172.21.15.7:0/3815323126 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb3c0001c00 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3cdd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3cdd9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3cdd9b640 10 monclient: dump: 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3d4bf6640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:15.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.445+0000 7fb3d4bf6640 4 set_mon_vals no callback set 2024-03-25T18:18:15.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.446+0000 7fb3d4bf6640 10 monclient: shutdown 2024-03-25T18:18:15.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.446+0000 7fb3d4bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:15.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.446+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3815323126 >> v1:172.21.15.7:6790/0 conn(0x7fb3d00b4ff0 legacy=0x7fb3d013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.446+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3815323126 shutdown_connections 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.446+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3815323126 >> 172.21.15.7:0/3815323126 conn(0x7fb3d00617a0 msgr2=0x7fb3d0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.446+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3815323126 shutdown_connections 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.446+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3815323126 wait complete. 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 10 monclient: build_initial_monmap 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 10 monclient: monmap: 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:15.449 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 1 librados: starting msgr at 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 1 librados: starting objecter 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 1 Processor -- start 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 1 -- start start 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 1 librados: setting wanted keys 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 1 librados: calling monclient init 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.448+0000 7fb3d4bf6640 10 monclient: init 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:15.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 10 monclient(hunting): picked mon.noname-c con 0x7fb3d00b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 10 monclient(hunting): picked mon.noname-b con 0x7fb3d00b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 10 monclient(hunting): picked mon.noname-a con 0x7fb3d013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00abe60 con 0x7fb3d013c480 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00ac140 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d00ac420 con 0x7fb3d00b4670 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:15.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.449+0000 7fb3d4bf6640 10 monclient(hunting): authenticate will time out at 7537.553993s 2024-03-25T18:18:15.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.460+0000 7fb3ce59c640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fb3d00b4670 0x7fb3d013b600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:48226/0 (socket says 172.21.15.7:48226) 2024-03-25T18:18:15.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.460+0000 7fb3ce59c640 1 -- 172.21.15.7:0/3505526753 learned_addr learned my addr 172.21.15.7:0/3505526753 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:15.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.466+0000 7fb3c6ffd640 10 client.?.objecter ms_handle_connect 0x7fb3d00b4670 2024-03-25T18:18:15.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.466+0000 7fb3c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:15.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.467+0000 7fb3c6ffd640 10 client.?.objecter ms_handle_connect 0x7fb3d00b4ff0 2024-03-25T18:18:15.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.467+0000 7fb3c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:15.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.467+0000 7fb3c6ffd640 10 client.?.objecter ms_handle_connect 0x7fb3d013c480 2024-03-25T18:18:15.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.467+0000 7fb3c6ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:15.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.473+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3313174494 0 0) 0x7fb3d00ac420 con 0x7fb3d00b4670 2024-03-25T18:18:15.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.473+0000 7fb3c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.473+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.473+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.473+0000 7fb3c6ffd640 10 monclient(hunting): my global_id is 4412 2024-03-25T18:18:15.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.473+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b0004eb0 con 0x7fb3d00b4670 2024-03-25T18:18:15.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.480+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3828918733 0 0) 0x7fb3d00abe60 con 0x7fb3d013c480 2024-03-25T18:18:15.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.480+0000 7fb3c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.480+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.480+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.480+0000 7fb3c6ffd640 10 monclient(hunting): my global_id is 4395 2024-03-25T18:18:15.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.481+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d00ac420 con 0x7fb3d013c480 2024-03-25T18:18:15.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.485+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2173703184 0 0) 0x7fb3d00ac140 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.485+0000 7fb3c6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:15.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.485+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:15.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.485+0000 7fb3c6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:15.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.485+0000 7fb3c6ffd640 10 monclient(hunting): my global_id is 4393 2024-03-25T18:18:15.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.485+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3d00abe60 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.499+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1662174265 0 0) 0x7fb3d00abe60 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.499+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3d00ac140 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.499+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb3b8001830 con 0x7fb3d00b4ff0 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.499+0000 7fb3c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.499+0000 7fb3c6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.499+0000 7fb3c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.499+0000 7fb3c6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:15.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.501+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3157005906 0 0) 0x7fb3b0004eb0 con 0x7fb3d00b4670 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.502+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3d00abe60 con 0x7fb3d00b4670 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.502+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb3c0001bc0 con 0x7fb3d00b4670 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.502+0000 7fb3c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.502+0000 7fb3c6ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.502+0000 7fb3c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:15.503 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 472676904 0 0) 0x7fb3d00ac420 con 0x7fb3d013c480 2024-03-25T18:18:15.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb3b0004eb0 con 0x7fb3d013c480 2024-03-25T18:18:15.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb3c8001b50 con 0x7fb3d013c480 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 10 monclient(hunting): dump: 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2103941274 0 0) 0x7fb3d00abe60 con 0x7fb3d00b4670 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 >> v1:172.21.15.183:6789/0 conn(0x7fb3d00b4ff0 legacy=0x7fb3d01381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 >> v1:172.21.15.7:6789/0 conn(0x7fb3d013c480 legacy=0x7fb3d01e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 monclient: found mon.c 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d013a8f0 con 0x7fb3d00b4670 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3c6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 5 monclient: authenticate success, global_id 4412 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 10 monclient: _renew_subs 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d00ac840 con 0x7fb3d00b4670 2024-03-25T18:18:15.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 10 client.4412.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 10 monclient: _renew_subs 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d01e3aa0 con 0x7fb3d00b4670 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.509+0000 7fb3d4bf6640 1 librados: init done 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.510+0000 7fb3d4bf6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.510+0000 7fb3d4bf6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.510+0000 7fb3d4bf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:15.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.510+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d006dbd0 con 0x7fb3d00b4670 2024-03-25T18:18:15.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.834+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb3c00011f0 con 0x7fb3d00b4670 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.834+0000 7fb3c6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.834+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb3c0003b00 con 0x7fb3d00b4670 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.834+0000 7fb3c6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.834+0000 7fb3c6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.834+0000 7fb3c6ffd640 10 monclient: dump: 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:15.836 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:15.837 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:15.837 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:15.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.834+0000 7fb3cd59a640 4 set_mon_vals no callback set 2024-03-25T18:18:15.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.835+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fb3c0003d20 con 0x7fb3d00b4670 2024-03-25T18:18:15.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.836+0000 7fb3c7fff640 10 monclient: get_auth_request con 0x7fb3b006e130 auth_method 0 2024-03-25T18:18:15.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.836+0000 7fb3c6ffd640 10 client.4412.objecter ms_handle_connect 0x7fb3b006e130 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fb3c0054870 con 0x7fb3d00b4670 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 10 client.4412.objecter ms_dispatch 0x7fb3d0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 3 client.4412.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 3 client.4412.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 20 client.4412.objecter dump_active .. 0 homeless 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb3c0050380 con 0x7fb3d00b4670 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:15.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:15.842+0000 7fb3c6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:16.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.109+0000 7fb3d4bf6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:16.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.109+0000 7fb3d4bf6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:16.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.109+0000 7fb3d4bf6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:16.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.109+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb3d014f100 con 0x7fb3d00b4670 2024-03-25T18:18:16.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.150+0000 7fb3c6ffd640 1 -- 172.21.15.7:0/3505526753 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fb3c00522b0 con 0x7fb3d00b4670 2024-03-25T18:18:16.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.150+0000 7fb3c6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:16.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.150+0000 7fb3c6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:16.151 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:16.151 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 10 librados: watch_flush enter 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 10 librados: watch_flush exit 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 20 client.4412.objecter shutdown clearing up homeless session... 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 10 client.4412.objecter successfully canceled tick 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 >> v1:172.21.15.7:6800/1717564767 conn(0x7fb3b006e130 legacy=0x7fb3b00705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 10 monclient: shutdown 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 >> v1:172.21.15.7:6790/0 conn(0x7fb3d00b4670 legacy=0x7fb3d013b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 shutdown_connections 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 >> 172.21.15.7:0/3505526753 conn(0x7fb3d00617a0 msgr2=0x7fb3d0140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.152+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 shutdown_connections 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.153+0000 7fb3d4bf6640 1 -- 172.21.15.7:0/3505526753 wait complete. 2024-03-25T18:18:16.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:16.153+0000 7fb3d4bf6640 1 librados: shutdown 2024-03-25T18:18:16.164 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:22.970 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:23.085 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.084+0000 7f1b73b3b640 10 monclient: get_monmap_and_config 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.084+0000 7f1b73b3b640 10 monclient: build_initial_monmap 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.084+0000 7f1b73b3b640 10 monclient: monmap: 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:23.085944+0000 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:23.085944+0000 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:23.086 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:23.087 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:23.087 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.085+0000 7f1b73b3b640 1 Processor -- start 2024-03-25T18:18:23.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.085+0000 7f1b73b3b640 1 -- start start 2024-03-25T18:18:23.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.085+0000 7f1b73b3b640 10 monclient: init 2024-03-25T18:18:23.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:23.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:23.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 10 monclient(hunting): picked mon.noname-c con 0x7f1b6c0b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:23.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 10 monclient(hunting): picked mon.noname-b con 0x7f1b6c0b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:23.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 10 monclient(hunting): picked mon.noname-a con 0x7f1b6c13c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:23.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0ab0e0 con 0x7f1b6c13c480 2024-03-25T18:18:23.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0ab3c0 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0a9f30 con 0x7f1b6c0b4670 2024-03-25T18:18:23.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:23.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.086+0000 7f1b73b3b640 10 monclient(hunting): authenticate will time out at 7545.191514s 2024-03-25T18:18:23.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.095+0000 7f1b710d6640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f1b6c0b4670 0x7f1b6c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:52126/0 (socket says 172.21.15.7:52126) 2024-03-25T18:18:23.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.095+0000 7f1b710d6640 1 -- 172.21.15.7:0/4285198683 learned_addr learned my addr 172.21.15.7:0/4285198683 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:23.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.103+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710235719 0 0) 0x7f1b6c0a9f30 con 0x7f1b6c0b4670 2024-03-25T18:18:23.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.103+0000 7f1b708d5640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.103+0000 7f1b708d5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.103+0000 7f1b708d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.103+0000 7f1b708d5640 10 monclient(hunting): my global_id is 4415 2024-03-25T18:18:23.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.103+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b58003520 con 0x7f1b6c0b4670 2024-03-25T18:18:23.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.112+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3746667746 0 0) 0x7f1b6c0ab0e0 con 0x7f1b6c13c480 2024-03-25T18:18:23.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.112+0000 7f1b708d5640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.112+0000 7f1b708d5640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.112+0000 7f1b708d5640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.112+0000 7f1b708d5640 10 monclient(hunting): my global_id is 4398 2024-03-25T18:18:23.114 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.112+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c0a9f30 con 0x7f1b6c13c480 2024-03-25T18:18:23.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 358623555 0 0) 0x7f1b58003520 con 0x7f1b6c0b4670 2024-03-25T18:18:23.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 >> v1:172.21.15.183:6789/0 conn(0x7f1b6c0b4ff0 legacy=0x7f1b6c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:23.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 >> v1:172.21.15.7:6789/0 conn(0x7f1b6c13c480 legacy=0x7f1b6c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 10 monclient: _finish_hunting 0 2024-03-25T18:18:23.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 1 monclient: found mon.noname-b 2024-03-25T18:18:23.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c0a9b70 con 0x7f1b6c0b4670 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 10 monclient: _finish_auth 0 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 10 monclient: _check_auth_tickets 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1b5c001690 con 0x7f1b6c0b4670 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b708d5640 10 monclient: dump: 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:23.126 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b73b3b640 5 monclient: authenticate success, global_id 4415 2024-03-25T18:18:23.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.124+0000 7f1b73b3b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:23.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b5c0017f0 con 0x7f1b6c0b4670 2024-03-25T18:18:23.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b708d5640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:23.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b708d5640 1 -- 172.21.15.7:0/4285198683 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1b5c001c00 con 0x7f1b6c0b4670 2024-03-25T18:18:23.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b708d5640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b708d5640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b708d5640 10 monclient: dump: 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b73b3b640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:23.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.129+0000 7f1b73b3b640 4 set_mon_vals no callback set 2024-03-25T18:18:23.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.130+0000 7f1b73b3b640 10 monclient: shutdown 2024-03-25T18:18:23.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.130+0000 7f1b73b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:23.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.130+0000 7f1b73b3b640 1 -- 172.21.15.7:0/4285198683 >> v1:172.21.15.7:6790/0 conn(0x7f1b6c0b4670 legacy=0x7f1b6c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.130+0000 7f1b73b3b640 1 -- 172.21.15.7:0/4285198683 shutdown_connections 2024-03-25T18:18:23.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.130+0000 7f1b73b3b640 1 -- 172.21.15.7:0/4285198683 >> 172.21.15.7:0/4285198683 conn(0x7f1b6c0617a0 msgr2=0x7f1b6c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:23.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.130+0000 7f1b73b3b640 1 -- 172.21.15.7:0/4285198683 shutdown_connections 2024-03-25T18:18:23.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.130+0000 7f1b73b3b640 1 -- 172.21.15.7:0/4285198683 wait complete. 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 10 monclient: build_initial_monmap 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 10 monclient: monmap: 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:23.135 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 1 librados: starting msgr at 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 1 librados: starting objecter 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 1 Processor -- start 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 1 -- start start 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 1 librados: setting wanted keys 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 1 librados: calling monclient init 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.131+0000 7f1b73b3b640 10 monclient: init 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 10 monclient(hunting): picked mon.noname-b con 0x7f1b6c0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 10 monclient(hunting): picked mon.noname-a con 0x7f1b6c0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 10 monclient(hunting): picked mon.noname-c con 0x7f1b6c13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c130c60 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c144800 con 0x7f1b6c0b4670 2024-03-25T18:18:23.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c144ae0 con 0x7f1b6c13c480 2024-03-25T18:18:23.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:23.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.132+0000 7f1b73b3b640 10 monclient(hunting): authenticate will time out at 7545.237359s 2024-03-25T18:18:23.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.145+0000 7f1b720d8640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f1b6c13c480 0x7f1b6c130550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:52180/0 (socket says 172.21.15.7:52180) 2024-03-25T18:18:23.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.145+0000 7f1b718d7640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1b6c0b4670 0x7f1b6c145270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:57960/0 (socket says 172.21.15.7:57960) 2024-03-25T18:18:23.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.145+0000 7f1b720d8640 1 -- 172.21.15.7:0/582249586 learned_addr learned my addr 172.21.15.7:0/582249586 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:23.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.146+0000 7f1b627fc640 10 client.?.objecter ms_handle_connect 0x7f1b6c13c480 2024-03-25T18:18:23.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.146+0000 7f1b627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:23.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.147+0000 7f1b627fc640 10 client.?.objecter ms_handle_connect 0x7f1b6c0b4670 2024-03-25T18:18:23.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.147+0000 7f1b627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:23.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.148+0000 7f1b627fc640 10 client.?.objecter ms_handle_connect 0x7f1b6c0b4ff0 2024-03-25T18:18:23.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.148+0000 7f1b627fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:23.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.160+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 414208039 0 0) 0x7f1b6c130c60 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.160+0000 7f1b627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.160+0000 7f1b627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.160+0000 7f1b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.160+0000 7f1b627fc640 10 monclient(hunting): my global_id is 4401 2024-03-25T18:18:23.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.160+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b50004eb0 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.163+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 372052830 0 0) 0x7f1b6c144800 con 0x7f1b6c0b4670 2024-03-25T18:18:23.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.163+0000 7f1b627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.163+0000 7f1b627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.163+0000 7f1b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.163+0000 7f1b627fc640 10 monclient(hunting): my global_id is 4396 2024-03-25T18:18:23.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.163+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c130c60 con 0x7f1b6c0b4670 2024-03-25T18:18:23.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.164+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389059466 0 0) 0x7f1b6c144ae0 con 0x7f1b6c13c480 2024-03-25T18:18:23.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.164+0000 7f1b627fc640 10 monclient(hunting): _negotiate 2024-03-25T18:18:23.165 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.164+0000 7f1b627fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:23.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.164+0000 7f1b627fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:23.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.164+0000 7f1b627fc640 10 monclient(hunting): my global_id is 4418 2024-03-25T18:18:23.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.164+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c144800 con 0x7f1b6c13c480 2024-03-25T18:18:23.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 575207173 0 0) 0x7f1b6c144800 con 0x7f1b6c13c480 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b6c144ae0 con 0x7f1b6c13c480 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1b64001c60 con 0x7f1b6c13c480 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 10 monclient(hunting): dump: 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.182 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:23.183 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:23.183 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:23.183 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:23.183 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.183 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:23.183 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1295225654 0 0) 0x7f1b50004eb0 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b6c144800 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1b5c001b00 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.181+0000 7f1b627fc640 10 monclient(hunting): dump: 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:23.184 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:23.185 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:23.185 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:23.185 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.189+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2328323909 0 0) 0x7f1b6c130c60 con 0x7f1b6c0b4670 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.189+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1b50004eb0 con 0x7f1b6c0b4670 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.189+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1b68003f40 con 0x7f1b6c0b4670 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.189+0000 7f1b627fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.189+0000 7f1b627fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.189+0000 7f1b627fc640 10 monclient(hunting): dump: 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:23.190 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:23.191 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3013834762 0 0) 0x7f1b6c144800 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 >> v1:172.21.15.7:6790/0 conn(0x7f1b6c13c480 legacy=0x7f1b6c130550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 >> v1:172.21.15.183:6789/0 conn(0x7f1b6c0b4670 legacy=0x7f1b6c145270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 10 monclient: _finish_hunting 0 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 1 monclient: found mon.a 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c1439a0 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 10 monclient: _finish_auth 0 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 10 monclient: _check_auth_tickets 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b627fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 5 monclient: authenticate success, global_id 4401 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 10 monclient: _renew_subs 2024-03-25T18:18:23.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c1463e0 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 10 client.4401.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 10 monclient: _renew_subs 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c143660 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.190+0000 7f1b73b3b640 1 librados: init done 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.191+0000 7f1b73b3b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.191+0000 7f1b73b3b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.191+0000 7f1b73b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:23.194 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.191+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c06dbd0 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.504+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b5c004a90 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.504+0000 7f1b627fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.504+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1b5c004da0 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.504+0000 7f1b627fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.504+0000 7f1b627fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.504+0000 7f1b627fc640 10 monclient: dump: 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:23.505 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:23.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.504+0000 7f1b63fff640 4 set_mon_vals no callback set 2024-03-25T18:18:23.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.505+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f1b5c01ad70 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.505+0000 7f1b718d7640 10 monclient: get_auth_request con 0x7f1b5006e130 auth_method 0 2024-03-25T18:18:23.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.506+0000 7f1b627fc640 10 client.4401.objecter ms_handle_connect 0x7f1b5006e130 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f1b5c054830 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 10 client.4401.objecter ms_dispatch 0x7f1b6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 3 client.4401.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 3 client.4401.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 20 client.4401.objecter dump_active .. 0 homeless 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b5c050340 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:23.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.511+0000 7f1b627fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:23.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.764+0000 7f1b73b3b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.764+0000 7f1b73b3b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.764+0000 7f1b73b3b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:23.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.764+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b6c13d940 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.798+0000 7f1b627fc640 1 -- 172.21.15.7:0/582249586 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f1b6c13d940 con 0x7f1b6c0b4ff0 2024-03-25T18:18:23.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.798+0000 7f1b627fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:23.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.798+0000 7f1b627fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:23.799 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:23.799 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 10 librados: watch_flush enter 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 10 librados: watch_flush exit 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 20 client.4401.objecter shutdown clearing up homeless session... 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 10 client.4401.objecter successfully canceled tick 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 >> v1:172.21.15.7:6800/1717564767 conn(0x7f1b5006e130 legacy=0x7f1b500705f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 10 monclient: shutdown 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:23.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.800+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 >> v1:172.21.15.7:6789/0 conn(0x7f1b6c0b4ff0 legacy=0x7f1b6c12fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:23.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.801+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 shutdown_connections 2024-03-25T18:18:23.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.801+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 >> 172.21.15.7:0/582249586 conn(0x7f1b6c0617a0 msgr2=0x7f1b6c140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:23.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.801+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 shutdown_connections 2024-03-25T18:18:23.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.801+0000 7f1b73b3b640 1 -- 172.21.15.7:0/582249586 wait complete. 2024-03-25T18:18:23.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:23.801+0000 7f1b73b3b640 1 librados: shutdown 2024-03-25T18:18:23.814 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:30.621 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:30.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.722+0000 7f8866431640 10 monclient: get_monmap_and_config 2024-03-25T18:18:30.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.722+0000 7f8866431640 10 monclient: build_initial_monmap 2024-03-25T18:18:30.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.722+0000 7f8866431640 10 monclient: monmap: 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:30.724604+0000 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:30.724604+0000 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:30.725 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:30.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.723+0000 7f8866431640 1 Processor -- start 2024-03-25T18:18:30.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.723+0000 7f8866431640 1 -- start start 2024-03-25T18:18:30.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.723+0000 7f8866431640 10 monclient: init 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 10 monclient(hunting): picked mon.noname-b con 0x7f88600b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 10 monclient(hunting): picked mon.noname-c con 0x7f88600b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 10 monclient(hunting): picked mon.noname-a con 0x7f886013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600ab0e0 con 0x7f886013c480 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600ab3c0 con 0x7f88600b4670 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88600a9f30 con 0x7f88600b4ff0 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:30.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.724+0000 7f8866431640 10 monclient(hunting): authenticate will time out at 7552.829993s 2024-03-25T18:18:30.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.735+0000 7f88649ce640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f886013c480 0x7f886014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55962/0 (socket says 172.21.15.7:55962) 2024-03-25T18:18:30.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.735+0000 7f88649ce640 1 -- 172.21.15.7:0/1875677355 learned_addr learned my addr 172.21.15.7:0/1875677355 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:30.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.743+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 230818758 0 0) 0x7f88600ab0e0 con 0x7f886013c480 2024-03-25T18:18:30.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.743+0000 7f885effd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.743+0000 7f885effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.743+0000 7f885effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.743+0000 7f885effd640 10 monclient(hunting): my global_id is 4404 2024-03-25T18:18:30.745 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.743+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8848003520 con 0x7f886013c480 2024-03-25T18:18:30.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.755+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3152980752 0 0) 0x7f8848003520 con 0x7f886013c480 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.755+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 >> v1:172.21.15.7:6790/0 conn(0x7f88600b4ff0 legacy=0x7f886013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.755+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 >> v1:172.21.15.183:6789/0 conn(0x7f88600b4670 legacy=0x7f88601425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 1 monclient: found mon.noname-a 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88600a9b70 con 0x7f886013c480 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 10 monclient: _finish_auth 0 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8858001a10 con 0x7f886013c480 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f885effd640 10 monclient: dump: 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f8866431640 5 monclient: authenticate success, global_id 4404 2024-03-25T18:18:30.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.756+0000 7f8866431640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8858001b70 con 0x7f886013c480 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f885effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f885effd640 1 -- 172.21.15.7:0/1875677355 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8858001f80 con 0x7f886013c480 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f885effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f885effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f885effd640 10 monclient: dump: 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.775 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.776 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:30.776 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:30.776 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:30.776 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:30.776 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:30.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f8866431640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:30.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.773+0000 7f8866431640 4 set_mon_vals no callback set 2024-03-25T18:18:30.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 10 monclient: shutdown 2024-03-25T18:18:30.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:30.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 1 -- 172.21.15.7:0/1875677355 >> v1:172.21.15.7:6789/0 conn(0x7f886013c480 legacy=0x7f886014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 1 -- 172.21.15.7:0/1875677355 shutdown_connections 2024-03-25T18:18:30.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 1 -- 172.21.15.7:0/1875677355 >> 172.21.15.7:0/1875677355 conn(0x7f88600617a0 msgr2=0x7f8860061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:30.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 1 -- 172.21.15.7:0/1875677355 shutdown_connections 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 1 -- 172.21.15.7:0/1875677355 wait complete. 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 10 monclient: build_initial_monmap 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.774+0000 7f8866431640 10 monclient: monmap: 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.775+0000 7f8866431640 1 librados: starting msgr at 2024-03-25T18:18:30.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.775+0000 7f8866431640 1 librados: starting objecter 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.775+0000 7f8866431640 1 Processor -- start 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.775+0000 7f8866431640 1 -- start start 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.775+0000 7f8866431640 1 librados: setting wanted keys 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.775+0000 7f8866431640 1 librados: calling monclient init 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.775+0000 7f8866431640 10 monclient: init 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 10 monclient(hunting): picked mon.noname-a con 0x7f88600b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 10 monclient(hunting): picked mon.noname-b con 0x7f88600b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 10 monclient(hunting): picked mon.noname-c con 0x7f886013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88601444e0 con 0x7f88600b4670 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88601dbf50 con 0x7f88600b4ff0 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8860144730 con 0x7f886013c480 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:30.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.776+0000 7f8866431640 10 monclient(hunting): authenticate will time out at 7552.882019s 2024-03-25T18:18:30.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.792+0000 7f885ffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f88600b4670 0x7f8860144e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55990/0 (socket says 172.21.15.7:55990) 2024-03-25T18:18:30.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.792+0000 7f885f7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f88600b4ff0 0x7f88601db130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:43932/0 (socket says 172.21.15.7:43932) 2024-03-25T18:18:30.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.792+0000 7f885ffff640 1 -- 172.21.15.7:0/4249580534 learned_addr learned my addr 172.21.15.7:0/4249580534 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:30.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.795+0000 7f885cff9640 10 client.?.objecter ms_handle_connect 0x7f88600b4ff0 2024-03-25T18:18:30.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.795+0000 7f885cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.795+0000 7f885cff9640 10 client.?.objecter ms_handle_connect 0x7f88600b4670 2024-03-25T18:18:30.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.795+0000 7f885cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.797+0000 7f885cff9640 10 client.?.objecter ms_handle_connect 0x7f886013c480 2024-03-25T18:18:30.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.797+0000 7f885cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:30.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.805+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 442911289 0 0) 0x7f8860144730 con 0x7f886013c480 2024-03-25T18:18:30.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.805+0000 7f885cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.805+0000 7f885cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.805+0000 7f885cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.805+0000 7f885cff9640 10 monclient(hunting): my global_id is 4424 2024-03-25T18:18:30.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.805+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f884c004eb0 con 0x7f886013c480 2024-03-25T18:18:30.810 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.809+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503242233 0 0) 0x7f88601dbf50 con 0x7f88600b4ff0 2024-03-25T18:18:30.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.809+0000 7f885cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.809+0000 7f885cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.809+0000 7f885cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.809+0000 7f885cff9640 10 monclient(hunting): my global_id is 4402 2024-03-25T18:18:30.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.809+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8860144730 con 0x7f88600b4ff0 2024-03-25T18:18:30.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.812+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2533981121 0 0) 0x7f88601444e0 con 0x7f88600b4670 2024-03-25T18:18:30.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.812+0000 7f885cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:30.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.812+0000 7f885cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:30.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.812+0000 7f885cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:30.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.812+0000 7f885cff9640 10 monclient(hunting): my global_id is 4407 2024-03-25T18:18:30.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.812+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88601dbf50 con 0x7f88600b4670 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.817+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3396138445 0 0) 0x7f8860144730 con 0x7f88600b4ff0 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.817+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88601444e0 con 0x7f88600b4ff0 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.817+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f88540018f0 con 0x7f88600b4ff0 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.817+0000 7f885cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.817+0000 7f885cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.817+0000 7f885cff9640 10 monclient(hunting): dump: 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:30.819 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:30.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.817+0000 7f885cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.822+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 679130761 0 0) 0x7f884c004eb0 con 0x7f886013c480 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.822+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8860144730 con 0x7f886013c480 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.822+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8858001e80 con 0x7f886013c480 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.822+0000 7f885cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.822+0000 7f885cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.822+0000 7f885cff9640 10 monclient(hunting): dump: 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:30.824 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:30.825 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:30.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.828+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4175536912 0 0) 0x7f88601dbf50 con 0x7f88600b4670 2024-03-25T18:18:30.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.828+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f884c004eb0 con 0x7f88600b4670 2024-03-25T18:18:30.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.828+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8850001970 con 0x7f88600b4670 2024-03-25T18:18:30.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.828+0000 7f885cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:30.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.828+0000 7f885cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.828+0000 7f885cff9640 10 monclient(hunting): dump: 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:30.831 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:30.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1312525836 0 0) 0x7f8860144730 con 0x7f886013c480 2024-03-25T18:18:30.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 >> v1:172.21.15.183:6789/0 conn(0x7f88600b4ff0 legacy=0x7f88601db130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 >> v1:172.21.15.7:6789/0 conn(0x7f88600b4670 legacy=0x7f8860144e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:30.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:30.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 1 monclient: found mon.c 2024-03-25T18:18:30.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8860144a10 con 0x7f886013c480 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f885cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 5 monclient: authenticate success, global_id 4424 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 10 monclient: _renew_subs 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88601465f0 con 0x7f886013c480 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 10 client.4424.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 10 monclient: _renew_subs 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f886013f6a0 con 0x7f886013c480 2024-03-25T18:18:30.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.830+0000 7f8866431640 1 librados: init done 2024-03-25T18:18:30.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.831+0000 7f8866431640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.831+0000 7f8866431640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:30.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.831+0000 7f8866431640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:30.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:30.831+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f886013f9a0 con 0x7f886013c480 2024-03-25T18:18:31.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.141+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8858002020 con 0x7f886013c480 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.141+0000 7f885cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.141+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8858004fb0 con 0x7f886013c480 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.141+0000 7f885cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.141+0000 7f885cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.141+0000 7f885cff9640 10 monclient: dump: 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:31.143 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:31.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.141+0000 7f885e7fc640 4 set_mon_vals no callback set 2024-03-25T18:18:31.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.142+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f885801af80 con 0x7f886013c480 2024-03-25T18:18:31.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.143+0000 7f885ffff640 10 monclient: get_auth_request con 0x7f884c06e0c0 auth_method 0 2024-03-25T18:18:31.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.143+0000 7f885cff9640 10 client.4424.objecter ms_handle_connect 0x7f884c06e0c0 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f8858054b10 con 0x7f886013c480 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 10 client.4424.objecter ms_dispatch 0x7f8860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 3 client.4424.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 3 client.4424.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 20 client.4424.objecter dump_active .. 0 homeless 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8858050620 con 0x7f886013c480 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:31.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.149+0000 7f885cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:31.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.394+0000 7f8866431640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:31.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.394+0000 7f8866431640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:31.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.394+0000 7f8866431640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:18:31.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.394+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f886006dbd0 con 0x7f886013c480 2024-03-25T18:18:31.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.424+0000 7f885cff9640 1 -- 172.21.15.7:0/4249580534 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f8858052550 con 0x7f886013c480 2024-03-25T18:18:31.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.424+0000 7f885cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:31.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.424+0000 7f885cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:31.425 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:31.425 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:31.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 10 librados: watch_flush enter 2024-03-25T18:18:31.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 10 librados: watch_flush exit 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 20 client.4424.objecter shutdown clearing up homeless session... 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 10 client.4424.objecter successfully canceled tick 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 >> v1:172.21.15.7:6800/1717564767 conn(0x7f884c06e0c0 legacy=0x7f884c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 10 monclient: shutdown 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 >> v1:172.21.15.7:6790/0 conn(0x7f886013c480 legacy=0x7f88601db840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 shutdown_connections 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 >> 172.21.15.7:0/4249580534 conn(0x7f88600617a0 msgr2=0x7f8860141a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 shutdown_connections 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 1 -- 172.21.15.7:0/4249580534 wait complete. 2024-03-25T18:18:31.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:31.426+0000 7f8866431640 1 librados: shutdown 2024-03-25T18:18:31.438 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:33.723 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:40.360 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.784 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:40.421 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.822 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:04:40.460 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.823 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:40.460 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.823 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:40.449 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.831 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:40.458 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.834 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:40.471 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.861 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:04:40.487 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:33.881 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:04:40.518 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:38.245 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.361+0000 7f6c2a9f4640 10 monclient: get_monmap_and_config 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.361+0000 7f6c2a9f4640 10 monclient: build_initial_monmap 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.361+0000 7f6c2a9f4640 10 monclient: monmap: 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:38.362595+0000 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:38.362595+0000 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:38.362 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:38.363 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:38.363 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:38.363 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:38.363 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.362+0000 7f6c2a9f4640 1 Processor -- start 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.362+0000 7f6c2a9f4640 1 -- start start 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.362+0000 7f6c2a9f4640 10 monclient: init 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c240b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c240b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c240b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240a96f0 con 0x7f6c240b1b00 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240ab0e0 con 0x7f6c240b4e70 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c24145f70 con 0x7f6c240b44b0 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:38.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.363+0000 7f6c2a9f4640 10 monclient(hunting): authenticate will time out at 7560.467964s 2024-03-25T18:18:38.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.371+0000 7f6c23fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f6c240b4e70 0x7f6c2413f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:48490/0 (socket says 172.21.15.7:48490) 2024-03-25T18:18:38.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.371+0000 7f6c23fff640 1 -- 172.21.15.7:0/286137642 learned_addr learned my addr 172.21.15.7:0/286137642 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:38.377 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.376+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1326704584 0 0) 0x7f6c240ab0e0 con 0x7f6c240b4e70 2024-03-25T18:18:38.377 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.376+0000 7f6c22ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.377 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.376+0000 7f6c22ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.377 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.376+0000 7f6c22ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.377 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.376+0000 7f6c22ffd640 10 monclient(hunting): my global_id is 4405 2024-03-25T18:18:38.377 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.376+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c0c003520 con 0x7f6c240b4e70 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.379+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3530518646 0 0) 0x7f6c0c003520 con 0x7f6c240b4e70 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.379+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 >> v1:172.21.15.7:6790/0 conn(0x7f6c240b44b0 legacy=0x7f6c24145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.379+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 >> v1:172.21.15.7:6789/0 conn(0x7f6c240b1b00 legacy=0x7f6c2414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.379+0000 7f6c22ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.379+0000 7f6c22ffd640 1 monclient: found mon.noname-c 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.379+0000 7f6c22ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240a9b80 con 0x7f6c240b4e70 2024-03-25T18:18:38.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 10 monclient: _finish_auth 0 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6c14001690 con 0x7f6c240b4e70 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c22ffd640 10 monclient: dump: 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:38.382 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c2a9f4640 5 monclient: authenticate success, global_id 4405 2024-03-25T18:18:38.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.380+0000 7f6c2a9f4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c14001860 con 0x7f6c240b4e70 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c22ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c22ffd640 1 -- 172.21.15.7:0/286137642 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6c14001cb0 con 0x7f6c240b4e70 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c22ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c22ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c22ffd640 10 monclient: dump: 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:38.385 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c2a9f4640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:38.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.384+0000 7f6c2a9f4640 4 set_mon_vals no callback set 2024-03-25T18:18:38.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.385+0000 7f6c2a9f4640 10 monclient: shutdown 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.385+0000 7f6c2a9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.385+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/286137642 >> v1:172.21.15.183:6789/0 conn(0x7f6c240b4e70 legacy=0x7f6c2413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.385+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/286137642 shutdown_connections 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.385+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/286137642 >> 172.21.15.7:0/286137642 conn(0x7f6c240617a0 msgr2=0x7f6c24061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.385+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/286137642 shutdown_connections 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.385+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/286137642 wait complete. 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.386+0000 7f6c2a9f4640 10 monclient: build_initial_monmap 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.386+0000 7f6c2a9f4640 10 monclient: monmap: 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:38.388 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.386+0000 7f6c2a9f4640 1 librados: starting msgr at 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.386+0000 7f6c2a9f4640 1 librados: starting objecter 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 1 Processor -- start 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 1 -- start start 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 1 librados: setting wanted keys 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 1 librados: calling monclient init 2024-03-25T18:18:38.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 10 monclient: init 2024-03-25T18:18:38.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:38.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:38.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 10 monclient(hunting): picked mon.noname-a con 0x7f6c240b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:38.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 10 monclient(hunting): picked mon.noname-c con 0x7f6c240b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:38.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.387+0000 7f6c2a9f4640 10 monclient(hunting): picked mon.noname-b con 0x7f6c240b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:38.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.388+0000 7f6c2a9f4640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240ae640 con 0x7f6c240b1b00 2024-03-25T18:18:38.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.388+0000 7f6c2a9f4640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240b1450 con 0x7f6c240b4e70 2024-03-25T18:18:38.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.388+0000 7f6c2a9f4640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240b1020 con 0x7f6c240b44b0 2024-03-25T18:18:38.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.388+0000 7f6c2a9f4640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:38.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.388+0000 7f6c2a9f4640 10 monclient(hunting): authenticate will time out at 7560.492670s 2024-03-25T18:18:38.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.396+0000 7f6c237fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f6c240b44b0 0x7f6c24138fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:42088/0 (socket says 172.21.15.7:42088) 2024-03-25T18:18:38.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.396+0000 7f6c237fe640 1 -- 172.21.15.7:0/4050798020 learned_addr learned my addr 172.21.15.7:0/4050798020 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:38.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.398+0000 7f6c20ff9640 10 client.?.objecter ms_handle_connect 0x7f6c240b44b0 2024-03-25T18:18:38.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.398+0000 7f6c20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:38.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.408+0000 7f6c20ff9640 10 client.?.objecter ms_handle_connect 0x7f6c240b4e70 2024-03-25T18:18:38.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.408+0000 7f6c20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:38.411 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.411+0000 7f6c20ff9640 10 client.?.objecter ms_handle_connect 0x7f6c240b1b00 2024-03-25T18:18:38.411 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.411+0000 7f6c20ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:38.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.413+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1150436796 0 0) 0x7f6c240b1020 con 0x7f6c240b44b0 2024-03-25T18:18:38.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.413+0000 7f6c20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.413+0000 7f6c20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.413+0000 7f6c20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.413+0000 7f6c20ff9640 10 monclient(hunting): my global_id is 4427 2024-03-25T18:18:38.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.413+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c10004eb0 con 0x7f6c240b44b0 2024-03-25T18:18:38.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.418+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3936594410 0 0) 0x7f6c240b1450 con 0x7f6c240b4e70 2024-03-25T18:18:38.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.418+0000 7f6c20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.418+0000 7f6c20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.418+0000 7f6c20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.418+0000 7f6c20ff9640 10 monclient(hunting): my global_id is 4408 2024-03-25T18:18:38.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.419+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c240b1020 con 0x7f6c240b4e70 2024-03-25T18:18:38.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.424+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4190538598 0 0) 0x7f6c240ae640 con 0x7f6c240b1b00 2024-03-25T18:18:38.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.424+0000 7f6c20ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:18:38.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.424+0000 7f6c20ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:38.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.424+0000 7f6c20ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:38.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.424+0000 7f6c20ff9640 10 monclient(hunting): my global_id is 4410 2024-03-25T18:18:38.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.424+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c240b1450 con 0x7f6c240b1b00 2024-03-25T18:18:38.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.426+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2926348473 0 0) 0x7f6c10004eb0 con 0x7f6c240b44b0 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.426+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c240ae640 con 0x7f6c240b44b0 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.427+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6c18001930 con 0x7f6c240b44b0 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.427+0000 7f6c20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.427+0000 7f6c20ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.427+0000 7f6c20ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.427+0000 7f6c20ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:38.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.432+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1373214863 0 0) 0x7f6c240b1020 con 0x7f6c240b4e70 2024-03-25T18:18:38.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.432+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c10004eb0 con 0x7f6c240b4e70 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.432+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6c1c001b10 con 0x7f6c240b4e70 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.432+0000 7f6c20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.432+0000 7f6c20ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.432+0000 7f6c20ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:38.434 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.435+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 782488145 0 0) 0x7f6c240b1450 con 0x7f6c240b1b00 2024-03-25T18:18:38.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.435+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6c240b1020 con 0x7f6c240b1b00 2024-03-25T18:18:38.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.435+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6c14001b70 con 0x7f6c240b1b00 2024-03-25T18:18:38.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.435+0000 7f6c20ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.435+0000 7f6c20ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.435+0000 7f6c20ff9640 10 monclient(hunting): dump: 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:38.437 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2380573199 0 0) 0x7f6c240b1020 con 0x7f6c240b1b00 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 >> v1:172.21.15.7:6790/0 conn(0x7f6c240b44b0 legacy=0x7f6c24138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 >> v1:172.21.15.183:6789/0 conn(0x7f6c240b4e70 legacy=0x7f6c241e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 1 monclient: found mon.a 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c240b11b0 con 0x7f6c240b1b00 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 10 monclient: _finish_auth 0 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c20ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 5 monclient: authenticate success, global_id 4410 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 10 monclient: _renew_subs 2024-03-25T18:18:38.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c241dac00 con 0x7f6c240b1b00 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 10 client.4410.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 10 monclient: _renew_subs 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c241db440 con 0x7f6c240b1b00 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.446+0000 7f6c2a9f4640 1 librados: init done 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.448+0000 7f6c2a9f4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.448+0000 7f6c2a9f4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.448+0000 7f6c2a9f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:38.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.448+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c2404f730 con 0x7f6c240b1b00 2024-03-25T18:18:38.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.748+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6c14001d50 con 0x7f6c240b1b00 2024-03-25T18:18:38.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.748+0000 7f6c20ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:38.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.748+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6c14004c30 con 0x7f6c240b1b00 2024-03-25T18:18:38.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.748+0000 7f6c20ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:38.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.748+0000 7f6c20ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.748+0000 7f6c20ff9640 10 monclient: dump: 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:38.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.748+0000 7f6c227fc640 4 set_mon_vals no callback set 2024-03-25T18:18:38.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.749+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f6c1401ac00 con 0x7f6c240b1b00 2024-03-25T18:18:38.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.750+0000 7f6c237fe640 10 monclient: get_auth_request con 0x7f6c1006e110 auth_method 0 2024-03-25T18:18:38.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.750+0000 7f6c20ff9640 10 client.4410.objecter ms_handle_connect 0x7f6c1006e110 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.757+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f6c14054790 con 0x7f6c240b1b00 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.757+0000 7f6c20ff9640 10 client.4410.objecter ms_dispatch 0x7f6c24000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.757+0000 7f6c20ff9640 3 client.4410.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.757+0000 7f6c20ff9640 3 client.4410.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.758+0000 7f6c20ff9640 20 client.4410.objecter dump_active .. 0 homeless 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.758+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6c140502a0 con 0x7f6c240b1b00 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.758+0000 7f6c20ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:38.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:38.758+0000 7f6c20ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:39.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.013+0000 7f6c2a9f4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:39.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.013+0000 7f6c2a9f4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:39.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.013+0000 7f6c2a9f4640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:18:39.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.013+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6c240aa090 con 0x7f6c240b1b00 2024-03-25T18:18:39.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.041+0000 7f6c20ff9640 1 -- 172.21.15.7:0/4050798020 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f6c140521d0 con 0x7f6c240b1b00 2024-03-25T18:18:39.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.041+0000 7f6c20ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:39.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.041+0000 7f6c20ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:39.042 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:39.042 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:39.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 10 librados: watch_flush enter 2024-03-25T18:18:39.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 10 librados: watch_flush exit 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 20 client.4410.objecter shutdown clearing up homeless session... 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 10 client.4410.objecter successfully canceled tick 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 >> v1:172.21.15.7:6800/1717564767 conn(0x7f6c1006e110 legacy=0x7f6c100705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 10 monclient: shutdown 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 >> v1:172.21.15.7:6789/0 conn(0x7f6c240b1b00 legacy=0x7f6c241434c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 shutdown_connections 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 >> 172.21.15.7:0/4050798020 conn(0x7f6c240617a0 msgr2=0x7f6c24061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 shutdown_connections 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.043+0000 7f6c2a9f4640 1 -- 172.21.15.7:0/4050798020 wait complete. 2024-03-25T18:18:39.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:39.044+0000 7f6c2a9f4640 1 librados: shutdown 2024-03-25T18:18:39.055 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:45.861 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:45.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.977+0000 7fce88f8d640 10 monclient: get_monmap_and_config 2024-03-25T18:18:45.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.977+0000 7fce88f8d640 10 monclient: build_initial_monmap 2024-03-25T18:18:45.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.977+0000 7fce88f8d640 10 monclient: monmap: 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:45.978709+0000 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:45.978709+0000 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:45.979 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:45.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.978+0000 7fce88f8d640 1 Processor -- start 2024-03-25T18:18:45.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.978+0000 7fce88f8d640 1 -- start start 2024-03-25T18:18:45.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.978+0000 7fce88f8d640 10 monclient: init 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 10 monclient(hunting): picked mon.noname-c con 0x7fce840b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 10 monclient(hunting): picked mon.noname-b con 0x7fce840b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 10 monclient(hunting): picked mon.noname-a con 0x7fce8413c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce840ab0e0 con 0x7fce8413c480 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce840ab3c0 con 0x7fce840b4ff0 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce840a9f30 con 0x7fce840b4670 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:45.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.979+0000 7fce88f8d640 10 monclient(hunting): authenticate will time out at 7568.083898s 2024-03-25T18:18:45.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.996+0000 7fce79d9b640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fce840b4670 0x7fce841425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:40676/0 (socket says 172.21.15.7:40676) 2024-03-25T18:18:45.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:45.996+0000 7fce79d9b640 1 -- 172.21.15.7:0/1514607173 learned_addr learned my addr 172.21.15.7:0/1514607173 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:46.010 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.009+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836535113 0 0) 0x7fce840a9f30 con 0x7fce840b4670 2024-03-25T18:18:46.010 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.009+0000 7fce81d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.010 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.009+0000 7fce81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.009+0000 7fce81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.009+0000 7fce81d9b640 10 monclient(hunting): my global_id is 4430 2024-03-25T18:18:46.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.009+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce70003520 con 0x7fce840b4670 2024-03-25T18:18:46.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.014+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1327282716 0 0) 0x7fce840ab0e0 con 0x7fce8413c480 2024-03-25T18:18:46.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.014+0000 7fce81d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.014+0000 7fce81d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.014+0000 7fce81d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.014+0000 7fce81d9b640 10 monclient(hunting): my global_id is 4413 2024-03-25T18:18:46.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.014+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce840a9f30 con 0x7fce8413c480 2024-03-25T18:18:46.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2893250441 0 0) 0x7fce840a9f30 con 0x7fce8413c480 2024-03-25T18:18:46.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 >> v1:172.21.15.7:6790/0 conn(0x7fce840b4670 legacy=0x7fce841425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 >> v1:172.21.15.183:6789/0 conn(0x7fce840b4ff0 legacy=0x7fce8413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:18:46.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 1 monclient: found mon.noname-a 2024-03-25T18:18:46.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce840a9b70 con 0x7fce8413c480 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 10 monclient: _finish_auth 0 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fce7c0019f0 con 0x7fce8413c480 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce88f8d640 5 monclient: authenticate success, global_id 4413 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce88f8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce81d9b640 10 monclient: dump: 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.027 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.028 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:46.028 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:46.028 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:46.028 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:46.028 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:46.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.025+0000 7fce88f8d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce7c001b50 con 0x7fce8413c480 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce81d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce81d9b640 1 -- 172.21.15.7:0/1514607173 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fce7c001f20 con 0x7fce8413c480 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce88f8d640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce88f8d640 4 set_mon_vals no callback set 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce81d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce81d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce81d9b640 10 monclient: dump: 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.039 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.040 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:46.040 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:46.040 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:46.040 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:46.040 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:46.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce88f8d640 10 monclient: shutdown 2024-03-25T18:18:46.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce88f8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:46.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.038+0000 7fce88f8d640 1 -- 172.21.15.7:0/1514607173 >> v1:172.21.15.7:6789/0 conn(0x7fce8413c480 legacy=0x7fce8414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.039+0000 7fce88f8d640 1 -- 172.21.15.7:0/1514607173 shutdown_connections 2024-03-25T18:18:46.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.039+0000 7fce88f8d640 1 -- 172.21.15.7:0/1514607173 >> 172.21.15.7:0/1514607173 conn(0x7fce840617a0 msgr2=0x7fce84061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:46.042 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.039+0000 7fce88f8d640 1 -- 172.21.15.7:0/1514607173 shutdown_connections 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.039+0000 7fce88f8d640 1 -- 172.21.15.7:0/1514607173 wait complete. 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.039+0000 7fce88f8d640 10 monclient: build_initial_monmap 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.039+0000 7fce88f8d640 10 monclient: monmap: 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.040+0000 7fce88f8d640 1 librados: starting msgr at 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.040+0000 7fce88f8d640 1 librados: starting objecter 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.040+0000 7fce88f8d640 1 Processor -- start 2024-03-25T18:18:46.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.040+0000 7fce88f8d640 1 -- start start 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.040+0000 7fce88f8d640 1 librados: setting wanted keys 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.040+0000 7fce88f8d640 1 librados: calling monclient init 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.040+0000 7fce88f8d640 10 monclient: init 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 10 monclient(hunting): picked mon.noname-a con 0x7fce840b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 10 monclient(hunting): picked mon.noname-c con 0x7fce840b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 10 monclient(hunting): picked mon.noname-b con 0x7fce8413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce8413c2f0 con 0x7fce840b4670 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce841447e0 con 0x7fce8413c480 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce84144ac0 con 0x7fce840b4ff0 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:46.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.041+0000 7fce88f8d640 10 monclient(hunting): authenticate will time out at 7568.146097s 2024-03-25T18:18:46.049 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.049+0000 7fce82d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fce8413c480 0x7fce841e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:52726/0 (socket says 172.21.15.7:52726) 2024-03-25T18:18:46.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.049+0000 7fce82d9d640 1 -- 172.21.15.7:0/1330352374 learned_addr learned my addr 172.21.15.7:0/1330352374 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:46.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.050+0000 7fce7b7fe640 10 client.?.objecter ms_handle_connect 0x7fce8413c480 2024-03-25T18:18:46.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.050+0000 7fce7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:46.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.062+0000 7fce7b7fe640 10 client.?.objecter ms_handle_connect 0x7fce840b4670 2024-03-25T18:18:46.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.063+0000 7fce7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:46.065 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.064+0000 7fce7b7fe640 10 client.?.objecter ms_handle_connect 0x7fce840b4ff0 2024-03-25T18:18:46.065 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.064+0000 7fce7b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:46.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.068+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3966155720 0 0) 0x7fce841447e0 con 0x7fce8413c480 2024-03-25T18:18:46.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.068+0000 7fce7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.068+0000 7fce7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.068+0000 7fce7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.068+0000 7fce7b7fe640 10 monclient(hunting): my global_id is 4414 2024-03-25T18:18:46.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.068+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce64004eb0 con 0x7fce8413c480 2024-03-25T18:18:46.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.076+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2879794744 0 0) 0x7fce84144ac0 con 0x7fce840b4ff0 2024-03-25T18:18:46.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.076+0000 7fce7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.076+0000 7fce7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.076+0000 7fce7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.076+0000 7fce7b7fe640 10 monclient(hunting): my global_id is 4433 2024-03-25T18:18:46.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.076+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce841447e0 con 0x7fce840b4ff0 2024-03-25T18:18:46.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.080+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1099287373 0 0) 0x7fce8413c2f0 con 0x7fce840b4670 2024-03-25T18:18:46.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.080+0000 7fce7b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:46.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.080+0000 7fce7b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:46.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.080+0000 7fce7b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:46.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.080+0000 7fce7b7fe640 10 monclient(hunting): my global_id is 4416 2024-03-25T18:18:46.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.080+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce84144ac0 con 0x7fce840b4670 2024-03-25T18:18:46.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.088+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 156789142 0 0) 0x7fce64004eb0 con 0x7fce8413c480 2024-03-25T18:18:46.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.088+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fce8413c2f0 con 0x7fce8413c480 2024-03-25T18:18:46.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.088+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fce7c001f60 con 0x7fce8413c480 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.088+0000 7fce7b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.088+0000 7fce7b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.088+0000 7fce7b7fe640 10 monclient(hunting): dump: 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:46.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.088+0000 7fce7b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 345947903 0 0) 0x7fce8413c2f0 con 0x7fce8413c480 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 >> v1:172.21.15.7:6790/0 conn(0x7fce840b4ff0 legacy=0x7fce84145f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 >> v1:172.21.15.7:6789/0 conn(0x7fce840b4670 legacy=0x7fce84145810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 1 monclient: found mon.b 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:46.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce84144da0 con 0x7fce8413c480 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce7b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 5 monclient: authenticate success, global_id 4414 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 10 monclient: _renew_subs 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce84145080 con 0x7fce8413c480 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 10 client.4414.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 10 monclient: _renew_subs 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fce84143f00 con 0x7fce8413c480 2024-03-25T18:18:46.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.103+0000 7fce88f8d640 1 librados: init done 2024-03-25T18:18:46.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.104+0000 7fce88f8d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:46.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.104+0000 7fce88f8d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:46.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.104+0000 7fce88f8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:46.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.104+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce840688c0 con 0x7fce8413c480 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.381+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fce7c003c20 con 0x7fce8413c480 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.381+0000 7fce7b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.381+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fce7c003fb0 con 0x7fce8413c480 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.381+0000 7fce7b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.381+0000 7fce7b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.381+0000 7fce7b7fe640 10 monclient: dump: 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:46.382 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:46.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.381+0000 7fce8159a640 4 set_mon_vals no callback set 2024-03-25T18:18:46.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.382+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fce7c0041d0 con 0x7fce8413c480 2024-03-25T18:18:46.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.382+0000 7fce8259c640 10 monclient: get_auth_request con 0x7fce6406de20 auth_method 0 2024-03-25T18:18:46.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.382+0000 7fce7b7fe640 10 client.4414.objecter ms_handle_connect 0x7fce6406de20 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fce7c054b40 con 0x7fce8413c480 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 10 client.4414.objecter ms_dispatch 0x7fce84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 3 client.4414.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 3 client.4414.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 20 client.4414.objecter dump_active .. 0 homeless 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fce7c050650 con 0x7fce8413c480 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:46.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.388+0000 7fce7b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:46.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.637+0000 7fce88f8d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.637+0000 7fce88f8d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.637+0000 7fce88f8d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:46.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.637+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fce8406dbd0 con 0x7fce8413c480 2024-03-25T18:18:46.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.736+0000 7fce7b7fe640 1 -- 172.21.15.7:0/1330352374 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fce7c052580 con 0x7fce8413c480 2024-03-25T18:18:46.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.736+0000 7fce7b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:46.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.736+0000 7fce7b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:46.737 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:46.737 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 10 librados: watch_flush enter 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 10 librados: watch_flush exit 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 20 client.4414.objecter shutdown clearing up homeless session... 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 10 client.4414.objecter successfully canceled tick 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 >> v1:172.21.15.7:6800/1717564767 conn(0x7fce6406de20 legacy=0x7fce640702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 10 monclient: shutdown 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 >> v1:172.21.15.183:6789/0 conn(0x7fce8413c480 legacy=0x7fce841e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 shutdown_connections 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.738+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 >> 172.21.15.7:0/1330352374 conn(0x7fce840617a0 msgr2=0x7fce841405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.739+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 shutdown_connections 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.739+0000 7fce88f8d640 1 -- 172.21.15.7:0/1330352374 wait complete. 2024-03-25T18:18:46.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:46.739+0000 7fce88f8d640 1 librados: shutdown 2024-03-25T18:18:46.751 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:18:53.557 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:18:53.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.665+0000 7ff1b1390640 10 monclient: get_monmap_and_config 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.665+0000 7ff1b1390640 10 monclient: build_initial_monmap 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.665+0000 7ff1b1390640 10 monclient: monmap: 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:18:53.666048+0000 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:18:53.666048+0000 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:18:53.666 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:53.667 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.665+0000 7ff1b1390640 1 Processor -- start 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.665+0000 7ff1b1390640 1 -- start start 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.665+0000 7ff1b1390640 10 monclient: init 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ac0b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ac0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ac0b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac0a96f0 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac0ab0e0 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac145f70 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:53.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.666+0000 7ff1b1390640 10 monclient(hunting): authenticate will time out at 7575.770911s 2024-03-25T18:18:53.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.682+0000 7ff1aaffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7ff1ac0b4e70 0x7ff1ac13f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:59198/0 (socket says 172.21.15.7:59198) 2024-03-25T18:18:53.683 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.682+0000 7ff1aaffd640 1 -- 172.21.15.7:0/3076785108 learned_addr learned my addr 172.21.15.7:0/3076785108 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:53.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.709+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511741303 0 0) 0x7ff1ac0a96f0 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.709+0000 7ff1a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.709+0000 7ff1a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.709+0000 7ff1a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.709+0000 7ff1a9ffb640 10 monclient(hunting): my global_id is 4419 2024-03-25T18:18:53.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.709+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff194003520 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.713+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4089384240 0 0) 0x7ff1ac0ab0e0 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.713+0000 7ff1a9ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.713+0000 7ff1a9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.713+0000 7ff1a9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.713+0000 7ff1a9ffb640 10 monclient(hunting): my global_id is 4417 2024-03-25T18:18:53.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.713+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ac0a96f0 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3758345391 0 0) 0x7ff1ac0a96f0 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 >> v1:172.21.15.7:6790/0 conn(0x7ff1ac0b44b0 legacy=0x7ff1ac145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 >> v1:172.21.15.7:6789/0 conn(0x7ff1ac0b4e70 legacy=0x7ff1ac13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:18:53.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 1 monclient: found mon.noname-c 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac0a9b80 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 10 monclient: _finish_auth 0 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff1a40019f0 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1b1390640 5 monclient: authenticate success, global_id 4417 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1b1390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1a9ffb640 10 monclient: dump: 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:53.725 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:53.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.723+0000 7ff1b1390640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1a4001b50 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1a9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1a9ffb640 1 -- 172.21.15.7:0/3076785108 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff1a4001fa0 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1b1390640 10 monclient: get_monmap_and_config success 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1b1390640 4 set_mon_vals no callback set 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1a9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1a9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.732+0000 7ff1a9ffb640 10 monclient: dump: 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.733 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.734 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.734 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:53.734 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:53.734 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:53.734 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:53.734 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 10 monclient: shutdown 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 1 -- 172.21.15.7:0/3076785108 >> v1:172.21.15.183:6789/0 conn(0x7ff1ac0b1b00 legacy=0x7ff1ac14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 1 -- 172.21.15.7:0/3076785108 shutdown_connections 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 1 -- 172.21.15.7:0/3076785108 >> 172.21.15.7:0/3076785108 conn(0x7ff1ac0617a0 msgr2=0x7ff1ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 1 -- 172.21.15.7:0/3076785108 shutdown_connections 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 1 -- 172.21.15.7:0/3076785108 wait complete. 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 10 monclient: build_initial_monmap 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.733+0000 7ff1b1390640 10 monclient: monmap: 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:18:53.736 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.734+0000 7ff1b1390640 1 librados: starting msgr at 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.734+0000 7ff1b1390640 1 librados: starting objecter 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.734+0000 7ff1b1390640 1 Processor -- start 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.734+0000 7ff1b1390640 1 -- start start 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.734+0000 7ff1b1390640 1 librados: setting wanted keys 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.734+0000 7ff1b1390640 1 librados: calling monclient init 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.734+0000 7ff1b1390640 10 monclient: init 2024-03-25T18:18:53.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 10 monclient: _reopen_session rank -1 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 10 monclient(hunting): picked mon.noname-c con 0x7ff1ac0b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 10 monclient(hunting): picked mon.noname-b con 0x7ff1ac0b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 10 monclient(hunting): picked mon.noname-a con 0x7ff1ac0b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac13be60 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac13c140 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1ac143200 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 10 monclient(hunting): _renew_subs 2024-03-25T18:18:53.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.735+0000 7ff1b1390640 10 monclient(hunting): authenticate will time out at 7575.839854s 2024-03-25T18:18:53.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.741+0000 7ff1aa7fc640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff1ac0b44b0 0x7ff1ac13db70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:59524/0 (socket says 172.21.15.7:59524) 2024-03-25T18:18:53.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.741+0000 7ff1aa7fc640 1 -- 172.21.15.7:0/2662070360 learned_addr learned my addr 172.21.15.7:0/2662070360 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:18:53.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.743+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff1ac0b44b0 2024-03-25T18:18:53.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.743+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.746+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff1ac0b4e70 2024-03-25T18:18:53.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.746+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.748+0000 7ff1937fe640 10 client.?.objecter ms_handle_connect 0x7ff1ac0b1b00 2024-03-25T18:18:53.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.748+0000 7ff1937fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:18:53.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.756+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 396299892 0 0) 0x7ff1ac13c140 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.756+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.756+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.756+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.756+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4420 2024-03-25T18:18:53.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.756+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff198004eb0 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.759+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320568526 0 0) 0x7ff1ac13be60 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.759+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.759+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.759+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.759+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4422 2024-03-25T18:18:53.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.759+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff1ac13c140 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.772+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3625879017 0 0) 0x7ff1ac13c140 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.772+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1ac13be60 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.772+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff1a4001e60 con 0x7ff1ac0b4e70 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.772+0000 7ff1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.772+0000 7ff1937fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.772+0000 7ff1937fe640 10 monclient(hunting): dump: 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:53.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.772+0000 7ff1937fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:18:53.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.776+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2700099038 0 0) 0x7ff198004eb0 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.777+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff1ac13c140 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.777+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff1a0001950 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.777+0000 7ff1937fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.777+0000 7ff1937fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.777+0000 7ff1937fe640 10 monclient(hunting): dump: 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:53.778 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:53.779 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:53.779 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:53.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.781+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1474798991 0 0) 0x7ff1ac143200 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.781+0000 7ff1937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:18:53.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.781+0000 7ff1937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:18:53.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.781+0000 7ff1937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:18:53.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.781+0000 7ff1937fe640 10 monclient(hunting): my global_id is 4439 2024-03-25T18:18:53.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.781+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff198004eb0 con 0x7ff1ac0b1b00 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.784+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1369065801 0 0) 0x7ff1ac13c140 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 >> v1:172.21.15.7:6790/0 conn(0x7ff1ac0b1b00 legacy=0x7ff1ac13d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 >> v1:172.21.15.7:6789/0 conn(0x7ff1ac0b4e70 legacy=0x7ff1ac1e1e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 1 monclient: found mon.b 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:53.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1ac13cb90 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 10 monclient: _finish_auth 0 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 5 monclient: authenticate success, global_id 4420 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 10 monclient: _renew_subs 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ac13c840 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 10 client.4420.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 10 monclient: _renew_subs 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ac13ae90 con 0x7ff1ac0b44b0 2024-03-25T18:18:53.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.785+0000 7ff1b1390640 1 librados: init done 2024-03-25T18:18:53.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.786+0000 7ff1b1390640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.786+0000 7ff1b1390640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:53.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.786+0000 7ff1b1390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:53.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:53.786+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ac04f730 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff1a00035f0 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff1a0003a00 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1937fe640 10 monclient: dump: 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:54.050 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1a97fa640 4 set_mon_vals no callback set 2024-03-25T18:18:54.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.049+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7ff1a0003c20 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.050+0000 7ff1aaffd640 10 monclient: get_auth_request con 0x7ff19806df40 auth_method 0 2024-03-25T18:18:54.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.050+0000 7ff1937fe640 10 client.4420.objecter ms_handle_connect 0x7ff19806df40 2024-03-25T18:18:54.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7ff1a0054800 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 10 client.4420.objecter ms_dispatch 0x7ff1ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:18:54.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 3 client.4420.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:18:54.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 3 client.4420.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:18:54.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 20 client.4420.objecter dump_active .. 0 homeless 2024-03-25T18:18:54.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff1a0050310 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:18:54.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.054+0000 7ff1937fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:18:54.131 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:00.768 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.140 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:00.777 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.166 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:00.803 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.186 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:00.812 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.189 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:00.816 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.195 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:00.832 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.198 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:00.836 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.217 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:00.843 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.236 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:00.874 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:18:54.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.282+0000 7ff1b1390640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:54.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.282+0000 7ff1b1390640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:54.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.282+0000 7ff1b1390640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:18:54.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.282+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff1ac13cd20 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.346+0000 7ff1937fe640 1 -- 172.21.15.7:0/2662070360 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7ff1a0052240 con 0x7ff1ac0b44b0 2024-03-25T18:18:54.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.346+0000 7ff1937fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:18:54.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.346+0000 7ff1937fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:18:54.347 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:18:54.347 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 10 librados: watch_flush enter 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 10 librados: watch_flush exit 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 20 client.4420.objecter shutdown clearing up homeless session... 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 10 client.4420.objecter successfully canceled tick 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 >> v1:172.21.15.7:6800/1717564767 conn(0x7ff19806df40 legacy=0x7ff198070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 10 monclient: shutdown 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.348+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 >> v1:172.21.15.183:6789/0 conn(0x7ff1ac0b44b0 legacy=0x7ff1ac13db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.349+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 shutdown_connections 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.349+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 >> 172.21.15.7:0/2662070360 conn(0x7ff1ac0617a0 msgr2=0x7ff1ac0b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.349+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 shutdown_connections 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.349+0000 7ff1b1390640 1 -- 172.21.15.7:0/2662070360 wait complete. 2024-03-25T18:18:54.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:18:54.349+0000 7ff1b1390640 1 librados: shutdown 2024-03-25T18:18:54.361 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:01.168 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:01.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.283+0000 7f491f486640 10 monclient: get_monmap_and_config 2024-03-25T18:19:01.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.283+0000 7f491f486640 10 monclient: build_initial_monmap 2024-03-25T18:19:01.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.283+0000 7f491f486640 10 monclient: monmap: 2024-03-25T18:19:01.284 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:01.284 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:01.284 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:01.284492+0000 2024-03-25T18:19:01.285 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:01.284492+0000 2024-03-25T18:19:01.285 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:01.285 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:01.285 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:01.285 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:01.285 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:01.285 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:01.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.284+0000 7f491f486640 1 Processor -- start 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.284+0000 7f491f486640 1 -- start start 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.284+0000 7f491f486640 10 monclient: init 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 10 monclient(hunting): picked mon.noname-a con 0x7f49180b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 10 monclient(hunting): picked mon.noname-b con 0x7f49180b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 10 monclient(hunting): picked mon.noname-c con 0x7f491813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180ab0e0 con 0x7f49180b4ff0 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180ab3c0 con 0x7f491813c480 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49180a9f30 con 0x7f49180b4670 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:01.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.285+0000 7f491f486640 10 monclient(hunting): authenticate will time out at 7583.389908s 2024-03-25T18:19:01.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.293+0000 7f491ca21640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f49180b4670 0x7f49181425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:56976/0 (socket says 172.21.15.7:56976) 2024-03-25T18:19:01.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.294+0000 7f491ca21640 1 -- 172.21.15.7:0/2371119542 learned_addr learned my addr 172.21.15.7:0/2371119542 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:01.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.311+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 29069648 0 0) 0x7f49180ab0e0 con 0x7f49180b4ff0 2024-03-25T18:19:01.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.311+0000 7f490ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.311+0000 7f490ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.311+0000 7f490ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.311+0000 7f490ffff640 10 monclient(hunting): my global_id is 4425 2024-03-25T18:19:01.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.311+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4904003520 con 0x7f49180b4ff0 2024-03-25T18:19:01.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.313+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1514153610 0 0) 0x7f49180a9f30 con 0x7f49180b4670 2024-03-25T18:19:01.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.313+0000 7f490ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.313+0000 7f490ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.313+0000 7f490ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.313+0000 7f490ffff640 10 monclient(hunting): my global_id is 4442 2024-03-25T18:19:01.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.313+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49180ab0e0 con 0x7f49180b4670 2024-03-25T18:19:01.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.319+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1583449989 0 0) 0x7f49180ab3c0 con 0x7f491813c480 2024-03-25T18:19:01.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.319+0000 7f490ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.319+0000 7f490ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.319+0000 7f490ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.319+0000 7f490ffff640 10 monclient(hunting): my global_id is 4423 2024-03-25T18:19:01.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.319+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f49180a9f30 con 0x7f491813c480 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1235311346 0 0) 0x7f4904003520 con 0x7f49180b4ff0 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 >> v1:172.21.15.7:6790/0 conn(0x7f49180b4670 legacy=0x7f49181425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 >> v1:172.21.15.183:6789/0 conn(0x7f491813c480 legacy=0x7f491814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 1 monclient: found mon.noname-a 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49180a9b70 con 0x7f49180b4ff0 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 10 monclient: _finish_auth 0 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4914001a10 con 0x7f49180b4ff0 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f491f486640 5 monclient: authenticate success, global_id 4425 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f491f486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:01.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f490ffff640 10 monclient: dump: 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:01.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.323+0000 7f491f486640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:01.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4914001b70 con 0x7f49180b4ff0 2024-03-25T18:19:01.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f490ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:01.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f490ffff640 1 -- 172.21.15.7:0/2371119542 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4914001f80 con 0x7f49180b4ff0 2024-03-25T18:19:01.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f490ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f490ffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f490ffff640 10 monclient: dump: 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f491f486640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:01.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f491f486640 4 set_mon_vals no callback set 2024-03-25T18:19:01.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f491f486640 10 monclient: shutdown 2024-03-25T18:19:01.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f491f486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:01.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f491f486640 1 -- 172.21.15.7:0/2371119542 >> v1:172.21.15.7:6789/0 conn(0x7f49180b4ff0 legacy=0x7f491813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f491f486640 1 -- 172.21.15.7:0/2371119542 shutdown_connections 2024-03-25T18:19:01.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.336+0000 7f491f486640 1 -- 172.21.15.7:0/2371119542 >> 172.21.15.7:0/2371119542 conn(0x7f49180617a0 msgr2=0x7f4918061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:01.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.337+0000 7f491f486640 1 -- 172.21.15.7:0/2371119542 shutdown_connections 2024-03-25T18:19:01.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.337+0000 7f491f486640 1 -- 172.21.15.7:0/2371119542 wait complete. 2024-03-25T18:19:01.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.337+0000 7f491f486640 10 monclient: build_initial_monmap 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.337+0000 7f491f486640 10 monclient: monmap: 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.338+0000 7f491f486640 1 librados: starting msgr at 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.338+0000 7f491f486640 1 librados: starting objecter 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.338+0000 7f491f486640 1 Processor -- start 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.338+0000 7f491f486640 1 -- start start 2024-03-25T18:19:01.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.338+0000 7f491f486640 1 librados: setting wanted keys 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.338+0000 7f491f486640 1 librados: calling monclient init 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.338+0000 7f491f486640 10 monclient: init 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 10 monclient(hunting): picked mon.noname-c con 0x7f49180b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 10 monclient(hunting): picked mon.noname-a con 0x7f49180b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 10 monclient(hunting): picked mon.noname-b con 0x7f491813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491813d0c0 con 0x7f49180b4ff0 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918130cc0 con 0x7f491813c480 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4918146010 con 0x7f49180b4670 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:01.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.339+0000 7f491f486640 10 monclient(hunting): authenticate will time out at 7583.444579s 2024-03-25T18:19:01.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.348+0000 7f491ca21640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f49180b4ff0 0x7f49181dbb90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:42490/0 (socket says 172.21.15.7:42490) 2024-03-25T18:19:01.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.348+0000 7f491ca21640 1 -- 172.21.15.7:0/872231553 learned_addr learned my addr 172.21.15.7:0/872231553 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:01.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.350+0000 7f490dffb640 10 client.?.objecter ms_handle_connect 0x7f49180b4670 2024-03-25T18:19:01.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.350+0000 7f490dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:01.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.351+0000 7f490dffb640 10 client.?.objecter ms_handle_connect 0x7f49180b4ff0 2024-03-25T18:19:01.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.351+0000 7f490dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:01.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.356+0000 7f490dffb640 10 client.?.objecter ms_handle_connect 0x7f491813c480 2024-03-25T18:19:01.356 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.356+0000 7f490dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 309054815 0 0) 0x7f4918146010 con 0x7f49180b4670 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): my global_id is 4445 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48fc004eb0 con 0x7f49180b4670 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1492233869 0 0) 0x7f491813d0c0 con 0x7f49180b4ff0 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 10 monclient(hunting): my global_id is 4428 2024-03-25T18:19:01.366 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.364+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4918146010 con 0x7f49180b4ff0 2024-03-25T18:19:01.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.367+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2590076849 0 0) 0x7f4918130cc0 con 0x7f491813c480 2024-03-25T18:19:01.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.367+0000 7f490dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:01.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.367+0000 7f490dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:01.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.367+0000 7f490dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:01.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.367+0000 7f490dffb640 10 monclient(hunting): my global_id is 4426 2024-03-25T18:19:01.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.367+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f491813d0c0 con 0x7f491813c480 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.381+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2925092079 0 0) 0x7f491813d0c0 con 0x7f491813c480 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.381+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4918130cc0 con 0x7f491813c480 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.381+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4910001850 con 0x7f491813c480 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.381+0000 7f490dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.381+0000 7f490dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.381+0000 7f490dffb640 10 monclient(hunting): dump: 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:01.382 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:01.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.381+0000 7f490dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:01.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.387+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4177077071 0 0) 0x7f4918146010 con 0x7f49180b4ff0 2024-03-25T18:19:01.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.387+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f491813d0c0 con 0x7f49180b4ff0 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.387+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4908001970 con 0x7f49180b4ff0 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.387+0000 7f490dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.387+0000 7f490dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.387+0000 7f490dffb640 10 monclient(hunting): dump: 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:01.389 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.390+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 109953444 0 0) 0x7f4918130cc0 con 0x7f491813c480 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 >> v1:172.21.15.7:6790/0 conn(0x7f49180b4670 legacy=0x7f49181db480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 >> v1:172.21.15.7:6789/0 conn(0x7f49180b4ff0 legacy=0x7f49181dbb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 1 monclient: found mon.b 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491813dbd0 con 0x7f491813c480 2024-03-25T18:19:01.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f490dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 5 monclient: authenticate success, global_id 4426 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 10 monclient: _renew_subs 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491813deb0 con 0x7f491813c480 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 10 client.4426.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 10 monclient: _renew_subs 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f491813e470 con 0x7f491813c480 2024-03-25T18:19:01.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.391+0000 7f491f486640 1 librados: init done 2024-03-25T18:19:01.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.392+0000 7f491f486640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.392+0000 7f491f486640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.392+0000 7f491f486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:01.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.392+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491813d270 con 0x7f491813c480 2024-03-25T18:19:01.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.673+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4910001c60 con 0x7f491813c480 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.674+0000 7f490dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.674+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4910003610 con 0x7f491813c480 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.674+0000 7f490dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.674+0000 7f490dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.674+0000 7f490dffb640 10 monclient: dump: 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:01.675 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:01.676 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:01.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.674+0000 7f490f7fe640 4 set_mon_vals no callback set 2024-03-25T18:19:01.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.674+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f49100195e0 con 0x7f491813c480 2024-03-25T18:19:01.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.675+0000 7f491d222640 10 monclient: get_auth_request con 0x7f48fc06df80 auth_method 0 2024-03-25T18:19:01.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.675+0000 7f490dffb640 10 client.4426.objecter ms_handle_connect 0x7f48fc06df80 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f4910053290 con 0x7f491813c480 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 10 client.4426.objecter ms_dispatch 0x7f4918000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 3 client.4426.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 3 client.4426.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 20 client.4426.objecter dump_active .. 0 homeless 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f491004eda0 con 0x7f491813c480 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:01.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.681+0000 7f490dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:01.920 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.919+0000 7f491f486640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.920 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.919+0000 7f491f486640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.920 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.919+0000 7f491f486640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:01.920 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.919+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f491806dbd0 con 0x7f491813c480 2024-03-25T18:19:01.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.944+0000 7f490dffb640 1 -- 172.21.15.7:0/872231553 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f4910050cd0 con 0x7f491813c480 2024-03-25T18:19:01.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.944+0000 7f490dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:01.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.944+0000 7f490dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:01.946 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:01.946 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:01.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 10 librados: watch_flush enter 2024-03-25T18:19:01.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 10 librados: watch_flush exit 2024-03-25T18:19:01.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 20 client.4426.objecter shutdown clearing up homeless session... 2024-03-25T18:19:01.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 10 client.4426.objecter successfully canceled tick 2024-03-25T18:19:01.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 >> v1:172.21.15.7:6800/1717564767 conn(0x7f48fc06df80 legacy=0x7f48fc070440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 10 monclient: shutdown 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 >> v1:172.21.15.183:6789/0 conn(0x7f491813c480 legacy=0x7f49181e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 shutdown_connections 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 >> 172.21.15.7:0/872231553 conn(0x7f49180617a0 msgr2=0x7f4918140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 shutdown_connections 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 1 -- 172.21.15.7:0/872231553 wait complete. 2024-03-25T18:19:01.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:01.947+0000 7f491f486640 1 librados: shutdown 2024-03-25T18:19:01.960 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:08.766 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:08.882 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.881+0000 7f303898f640 10 monclient: get_monmap_and_config 2024-03-25T18:19:08.882 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.881+0000 7f303898f640 10 monclient: build_initial_monmap 2024-03-25T18:19:08.882 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.881+0000 7f303898f640 10 monclient: monmap: 2024-03-25T18:19:08.882 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:08.882 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:08.882 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:08.882282+0000 2024-03-25T18:19:08.883 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:08.882282+0000 2024-03-25T18:19:08.883 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:08.883 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:08.883 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:08.883 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:08.883 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:08.883 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.882+0000 7f303898f640 1 Processor -- start 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.882+0000 7f303898f640 1 -- start start 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.882+0000 7f303898f640 10 monclient: init 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.882+0000 7f303898f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.882+0000 7f303898f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.882+0000 7f303898f640 10 monclient(hunting): picked mon.noname-a con 0x7f30340b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.883+0000 7f303898f640 10 monclient(hunting): picked mon.noname-b con 0x7f30340b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:08.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.883+0000 7f303898f640 10 monclient(hunting): picked mon.noname-c con 0x7f303413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:08.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.883+0000 7f303898f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30340ab0e0 con 0x7f30340b4ff0 2024-03-25T18:19:08.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.883+0000 7f303898f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30340ab3c0 con 0x7f303413c480 2024-03-25T18:19:08.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.883+0000 7f303898f640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30340a9f30 con 0x7f30340b4670 2024-03-25T18:19:08.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.883+0000 7f303898f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:08.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.883+0000 7f303898f640 10 monclient(hunting): authenticate will time out at 7590.987763s 2024-03-25T18:19:08.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.896+0000 7f3032d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f303413c480 0x7f303414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:41222/0 (socket says 172.21.15.7:41222) 2024-03-25T18:19:08.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.896+0000 7f3032d9d640 1 -- 172.21.15.7:0/1155825374 learned_addr learned my addr 172.21.15.7:0/1155825374 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:08.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.912+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1633612119 0 0) 0x7f30340ab3c0 con 0x7f303413c480 2024-03-25T18:19:08.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.912+0000 7f3031d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.912+0000 7f3031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.912+0000 7f3031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.914 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.912+0000 7f3031d9b640 10 monclient(hunting): my global_id is 4429 2024-03-25T18:19:08.914 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.912+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3020003520 con 0x7f303413c480 2024-03-25T18:19:08.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.922+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1845863898 0 0) 0x7f30340a9f30 con 0x7f30340b4670 2024-03-25T18:19:08.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.922+0000 7f3031d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.922+0000 7f3031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.922+0000 7f3031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.922+0000 7f3031d9b640 10 monclient(hunting): my global_id is 4448 2024-03-25T18:19:08.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.922+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30340ab3c0 con 0x7f30340b4670 2024-03-25T18:19:08.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.928+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 579118901 0 0) 0x7f30340ab0e0 con 0x7f30340b4ff0 2024-03-25T18:19:08.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.928+0000 7f3031d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.928+0000 7f3031d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.928+0000 7f3031d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.928+0000 7f3031d9b640 10 monclient(hunting): my global_id is 4431 2024-03-25T18:19:08.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.928+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30340a9f30 con 0x7f30340b4ff0 2024-03-25T18:19:08.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3141783685 0 0) 0x7f30340ab3c0 con 0x7f30340b4670 2024-03-25T18:19:08.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 >> v1:172.21.15.183:6789/0 conn(0x7f303413c480 legacy=0x7f303414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 >> v1:172.21.15.7:6789/0 conn(0x7f30340b4ff0 legacy=0x7f303413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 1 monclient: found mon.noname-b 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30340a9b70 con 0x7f30340b4670 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 10 monclient: _finish_auth 0 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f301c001670 con 0x7f30340b4670 2024-03-25T18:19:08.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f3031d9b640 10 monclient: dump: 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:08.935 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:08.936 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:08.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f303898f640 5 monclient: authenticate success, global_id 4448 2024-03-25T18:19:08.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.932+0000 7f303898f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:08.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f301c0017d0 con 0x7f30340b4670 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f3031d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f3031d9b640 1 -- 172.21.15.7:0/1155825374 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f301c001be0 con 0x7f30340b4670 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f303898f640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f303898f640 4 set_mon_vals no callback set 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f3031d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f3031d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.939+0000 7f3031d9b640 10 monclient: dump: 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.941 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.942 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.942 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:08.942 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:08.942 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:08.942 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:08.942 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:08.944 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.940+0000 7f303898f640 10 monclient: shutdown 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.940+0000 7f303898f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.940+0000 7f303898f640 1 -- 172.21.15.7:0/1155825374 >> v1:172.21.15.7:6790/0 conn(0x7f30340b4670 legacy=0x7f30341425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.940+0000 7f303898f640 1 -- 172.21.15.7:0/1155825374 shutdown_connections 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.940+0000 7f303898f640 1 -- 172.21.15.7:0/1155825374 >> 172.21.15.7:0/1155825374 conn(0x7f30340617a0 msgr2=0x7f3034061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.940+0000 7f303898f640 1 -- 172.21.15.7:0/1155825374 shutdown_connections 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.940+0000 7f303898f640 1 -- 172.21.15.7:0/1155825374 wait complete. 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.941+0000 7f303898f640 10 monclient: build_initial_monmap 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.941+0000 7f303898f640 10 monclient: monmap: 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:08.945 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 librados: starting msgr at 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 librados: starting objecter 2024-03-25T18:19:08.946 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 Processor -- start 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 -- start start 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 librados: setting wanted keys 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 librados: calling monclient init 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient: init 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient(hunting): picked mon.noname-c con 0x7f30340b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:08.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient(hunting): picked mon.noname-b con 0x7f30340b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:08.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient(hunting): picked mon.noname-a con 0x7f303413c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:08.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f303413c2b0 con 0x7f303413c480 2024-03-25T18:19:08.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3034130ad0 con 0x7f30340b4ff0 2024-03-25T18:19:08.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30341456d0 con 0x7f30340b4670 2024-03-25T18:19:08.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:08.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.942+0000 7f303898f640 10 monclient(hunting): authenticate will time out at 7591.047515s 2024-03-25T18:19:08.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.955+0000 7f302bfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f30340b4ff0 0x7f303412fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:41268/0 (socket says 172.21.15.7:41268) 2024-03-25T18:19:08.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.955+0000 7f302bfff640 1 -- 172.21.15.7:0/3411992569 learned_addr learned my addr 172.21.15.7:0/3411992569 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:08.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.955+0000 7f302affd640 10 client.?.objecter ms_handle_connect 0x7f30340b4ff0 2024-03-25T18:19:08.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.955+0000 7f302affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.958+0000 7f302affd640 10 client.?.objecter ms_handle_connect 0x7f30340b4670 2024-03-25T18:19:08.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.958+0000 7f302affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.962+0000 7f302affd640 10 client.?.objecter ms_handle_connect 0x7f303413c480 2024-03-25T18:19:08.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.962+0000 7f302affd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:08.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.967+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3039107983 0 0) 0x7f30341456d0 con 0x7f30340b4670 2024-03-25T18:19:08.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.967+0000 7f302affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.967+0000 7f302affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.968 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.967+0000 7f302affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.967+0000 7f302affd640 10 monclient(hunting): my global_id is 4451 2024-03-25T18:19:08.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.967+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3014004eb0 con 0x7f30340b4670 2024-03-25T18:19:08.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.973+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119424929 0 0) 0x7f3034130ad0 con 0x7f30340b4ff0 2024-03-25T18:19:08.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.973+0000 7f302affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.973+0000 7f302affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.973+0000 7f302affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.973+0000 7f302affd640 10 monclient(hunting): my global_id is 4432 2024-03-25T18:19:08.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.973+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30341456d0 con 0x7f30340b4ff0 2024-03-25T18:19:08.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.979+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 142715180 0 0) 0x7f303413c2b0 con 0x7f303413c480 2024-03-25T18:19:08.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.979+0000 7f302affd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:08.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.979+0000 7f302affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:08.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.979+0000 7f302affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:08.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.979+0000 7f302affd640 10 monclient(hunting): my global_id is 4434 2024-03-25T18:19:08.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.979+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3034130ad0 con 0x7f303413c480 2024-03-25T18:19:08.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.995+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 576667913 0 0) 0x7f3034130ad0 con 0x7f303413c480 2024-03-25T18:19:08.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.995+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f303413c2b0 con 0x7f303413c480 2024-03-25T18:19:08.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.995+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f302c001a30 con 0x7f303413c480 2024-03-25T18:19:08.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.995+0000 7f302affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.995+0000 7f302affd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.995+0000 7f302affd640 10 monclient(hunting): dump: 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:08.997 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:08.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.995+0000 7f302affd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:08.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.996+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3164056682 0 0) 0x7f30341456d0 con 0x7f30340b4ff0 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.996+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3034130ad0 con 0x7f30340b4ff0 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.996+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f301c001670 con 0x7f30340b4ff0 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.996+0000 7f302affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.996+0000 7f302affd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:08.996+0000 7f302affd640 10 monclient(hunting): dump: 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:08.999 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.000 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:09.000 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:09.000 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:09.000 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:09.000 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.002+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2446590654 0 0) 0x7f3014004eb0 con 0x7f30340b4670 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.003+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f30341456d0 con 0x7f30340b4670 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.003+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f30240018f0 con 0x7f30340b4670 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.003+0000 7f302affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.003+0000 7f302affd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:09.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.003+0000 7f302affd640 10 monclient(hunting): dump: 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:09.005 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:09.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4288122328 0 0) 0x7f30341456d0 con 0x7f30340b4670 2024-03-25T18:19:09.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 >> v1:172.21.15.183:6789/0 conn(0x7f30340b4ff0 legacy=0x7f303412fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 >> v1:172.21.15.7:6789/0 conn(0x7f303413c480 legacy=0x7f30341303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:09.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 1 monclient: found mon.c 2024-03-25T18:19:09.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3034144450 con 0x7f30340b4670 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 10 monclient: _finish_auth 0 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f302affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 5 monclient: authenticate success, global_id 4451 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 10 monclient: _renew_subs 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:09.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3034144730 con 0x7f30340b4670 2024-03-25T18:19:09.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 10 client.4451.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:09.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 10 monclient: _renew_subs 2024-03-25T18:19:09.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:09.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f303413ec50 con 0x7f30340b4670 2024-03-25T18:19:09.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.019+0000 7f303898f640 1 librados: init done 2024-03-25T18:19:09.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.020+0000 7f303898f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:09.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.020+0000 7f303898f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:09.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.020+0000 7f303898f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:09.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.020+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303406dbd0 con 0x7f30340b4670 2024-03-25T18:19:09.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.334+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3024003610 con 0x7f30340b4670 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.334+0000 7f302affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.334+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3024003920 con 0x7f30340b4670 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.334+0000 7f302affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.334+0000 7f302affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.334+0000 7f302affd640 10 monclient: dump: 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:09.336 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:09.337 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:09.337 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:09.337 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:09.337 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:09.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.334+0000 7f303159a640 4 set_mon_vals no callback set 2024-03-25T18:19:09.337 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.335+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f30240198f0 con 0x7f30340b4670 2024-03-25T18:19:09.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.335+0000 7f302bfff640 10 monclient: get_auth_request con 0x7f301406e160 auth_method 0 2024-03-25T18:19:09.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.335+0000 7f302affd640 10 client.4451.objecter ms_handle_connect 0x7f301406e160 2024-03-25T18:19:09.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f30240535c0 con 0x7f30340b4670 2024-03-25T18:19:09.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 10 client.4451.objecter ms_dispatch 0x7f3034000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:09.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 3 client.4451.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:09.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 3 client.4451.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:09.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 20 client.4451.objecter dump_active .. 0 homeless 2024-03-25T18:19:09.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f302404ef00 con 0x7f30340b4670 2024-03-25T18:19:09.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:09.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.341+0000 7f302affd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:09.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.603+0000 7f303898f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.603+0000 7f303898f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.603+0000 7f303898f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:09.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.603+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f30340688c0 con 0x7f30340b4670 2024-03-25T18:19:09.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.627+0000 7f302affd640 1 -- 172.21.15.7:0/3411992569 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f3024051000 con 0x7f30340b4670 2024-03-25T18:19:09.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.627+0000 7f302affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:09.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.627+0000 7f302affd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:09.628 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:09.629 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:09.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 10 librados: watch_flush enter 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 10 librados: watch_flush exit 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 20 client.4451.objecter shutdown clearing up homeless session... 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 10 client.4451.objecter successfully canceled tick 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 >> v1:172.21.15.7:6800/1717564767 conn(0x7f301406e160 legacy=0x7f3014070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 10 monclient: shutdown 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 >> v1:172.21.15.7:6790/0 conn(0x7f30340b4670 legacy=0x7f3034145bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 shutdown_connections 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 >> 172.21.15.7:0/3411992569 conn(0x7f30340617a0 msgr2=0x7f3034140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 shutdown_connections 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 1 -- 172.21.15.7:0/3411992569 wait complete. 2024-03-25T18:19:09.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:09.630+0000 7f303898f640 1 librados: shutdown 2024-03-25T18:19:09.644 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:13.935 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:20.572 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.946 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:20.583 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.985 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:20.623 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.990 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:20.627 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:13.998 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:20.624 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.003 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:20.640 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.004 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:20.631 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.030 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:20.656 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:14.036 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:20.673 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:16.450 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.550+0000 7fa8c0123640 10 monclient: get_monmap_and_config 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.550+0000 7fa8c0123640 10 monclient: build_initial_monmap 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.550+0000 7fa8c0123640 10 monclient: monmap: 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:16.551295+0000 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:16.551295+0000 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:16.551 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:16.552 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:16.552 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:16.552 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.550+0000 7fa8c0123640 1 Processor -- start 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 1 -- start start 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 10 monclient: init 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b80b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b80b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b813c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b80ab0e0 con 0x7fa8b80b4670 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b80ab3c0 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.553 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.551+0000 7fa8c0123640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b80a9f30 con 0x7fa8b813c480 2024-03-25T18:19:16.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.552+0000 7fa8c0123640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:16.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.552+0000 7fa8c0123640 10 monclient(hunting): authenticate will time out at 7598.656642s 2024-03-25T18:19:16.568 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.568+0000 7fa8be6c0640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fa8b813c480 0x7fa8b814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:43714/0 (socket says 172.21.15.7:43714) 2024-03-25T18:19:16.569 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.568+0000 7fa8be6c0640 1 -- 172.21.15.7:0/2115019723 learned_addr learned my addr 172.21.15.7:0/2115019723 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:16.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.577+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 617225822 0 0) 0x7fa8b80ab3c0 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.577+0000 7fa8bcebd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.577+0000 7fa8bcebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.577+0000 7fa8bcebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.577+0000 7fa8bcebd640 10 monclient(hunting): my global_id is 4435 2024-03-25T18:19:16.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.577+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a4003520 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.581+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1840103236 0 0) 0x7fa8b80a9f30 con 0x7fa8b813c480 2024-03-25T18:19:16.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.581+0000 7fa8bcebd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.581+0000 7fa8bcebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.581+0000 7fa8bcebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.581+0000 7fa8bcebd640 10 monclient(hunting): my global_id is 4454 2024-03-25T18:19:16.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.581+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b80ab3c0 con 0x7fa8b813c480 2024-03-25T18:19:16.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.590+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 702258527 0 0) 0x7fa8b80ab0e0 con 0x7fa8b80b4670 2024-03-25T18:19:16.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.590+0000 7fa8bcebd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.590+0000 7fa8bcebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.590+0000 7fa8bcebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.590+0000 7fa8bcebd640 10 monclient(hunting): my global_id is 4437 2024-03-25T18:19:16.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.590+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b80a9f30 con 0x7fa8b80b4670 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3287611452 0 0) 0x7fa8a4003520 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 >> v1:172.21.15.7:6790/0 conn(0x7fa8b813c480 legacy=0x7fa8b814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 >> v1:172.21.15.7:6789/0 conn(0x7fa8b80b4670 legacy=0x7fa8b81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 1 monclient: found mon.noname-c 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b80a9b70 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 10 monclient: _finish_auth 0 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa8b4001690 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8bcebd640 10 monclient: dump: 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:16.595 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:16.596 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:16.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8c0123640 5 monclient: authenticate success, global_id 4435 2024-03-25T18:19:16.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.593+0000 7fa8c0123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:16.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.599+0000 7fa8bcebd640 1 -- 172.21.15.7:0/2115019723 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8b40017f0 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.599+0000 7fa8bcebd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:16.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.599+0000 7fa8c0123640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:16.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.599+0000 7fa8c0123640 4 set_mon_vals no callback set 2024-03-25T18:19:16.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.600+0000 7fa8c0123640 10 monclient: shutdown 2024-03-25T18:19:16.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.600+0000 7fa8c0123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:16.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.600+0000 7fa8c0123640 1 -- 172.21.15.7:0/2115019723 >> v1:172.21.15.183:6789/0 conn(0x7fa8b80b4ff0 legacy=0x7fa8b813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.600+0000 7fa8c0123640 1 -- 172.21.15.7:0/2115019723 shutdown_connections 2024-03-25T18:19:16.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.600+0000 7fa8c0123640 1 -- 172.21.15.7:0/2115019723 >> 172.21.15.7:0/2115019723 conn(0x7fa8b80617a0 msgr2=0x7fa8b8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:16.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.600+0000 7fa8c0123640 1 -- 172.21.15.7:0/2115019723 shutdown_connections 2024-03-25T18:19:16.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.600+0000 7fa8c0123640 1 -- 172.21.15.7:0/2115019723 wait complete. 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.601+0000 7fa8c0123640 10 monclient: build_initial_monmap 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.601+0000 7fa8c0123640 10 monclient: monmap: 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:16.602 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:16.603 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:16.603 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:16.603 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.601+0000 7fa8c0123640 1 librados: starting msgr at 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.601+0000 7fa8c0123640 1 librados: starting objecter 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 1 Processor -- start 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 1 -- start start 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 1 librados: setting wanted keys 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 1 librados: calling monclient init 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 10 monclient: init 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:16.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 10 monclient(hunting): picked mon.noname-c con 0x7fa8b80b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:16.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.602+0000 7fa8c0123640 10 monclient(hunting): picked mon.noname-b con 0x7fa8b80b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:16.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.603+0000 7fa8c0123640 10 monclient(hunting): picked mon.noname-a con 0x7fa8b813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:16.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.603+0000 7fa8c0123640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b8144070 con 0x7fa8b813c480 2024-03-25T18:19:16.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.603+0000 7fa8c0123640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b81442c0 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.603+0000 7fa8c0123640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8b81445a0 con 0x7fa8b80b4670 2024-03-25T18:19:16.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.603+0000 7fa8c0123640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:16.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.603+0000 7fa8c0123640 10 monclient(hunting): authenticate will time out at 7598.707721s 2024-03-25T18:19:16.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.616+0000 7fa8bdebf640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fa8b80b4670 0x7fa8b8144ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:43738/0 (socket says 172.21.15.7:43738) 2024-03-25T18:19:16.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.616+0000 7fa8bdebf640 1 -- 172.21.15.7:0/4250834290 learned_addr learned my addr 172.21.15.7:0/4250834290 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:16.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.618+0000 7fa8ae7fc640 10 client.?.objecter ms_handle_connect 0x7fa8b813c480 2024-03-25T18:19:16.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.618+0000 7fa8ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.619+0000 7fa8ae7fc640 10 client.?.objecter ms_handle_connect 0x7fa8b80b4670 2024-03-25T18:19:16.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.619+0000 7fa8ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.623 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.623+0000 7fa8ae7fc640 10 client.?.objecter ms_handle_connect 0x7fa8b80b4ff0 2024-03-25T18:19:16.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.623+0000 7fa8ae7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:16.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.629+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100662695 0 0) 0x7fa8b8144070 con 0x7fa8b813c480 2024-03-25T18:19:16.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.629+0000 7fa8ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.629+0000 7fa8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.629+0000 7fa8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.629+0000 7fa8ae7fc640 10 monclient(hunting): my global_id is 4440 2024-03-25T18:19:16.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.629+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa89c004eb0 con 0x7fa8b813c480 2024-03-25T18:19:16.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.630+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 607540361 0 0) 0x7fa8b81442c0 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.630+0000 7fa8ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.630+0000 7fa8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.630+0000 7fa8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.630+0000 7fa8ae7fc640 10 monclient(hunting): my global_id is 4438 2024-03-25T18:19:16.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.630+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b8144070 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.632+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1444233621 0 0) 0x7fa8b81445a0 con 0x7fa8b80b4670 2024-03-25T18:19:16.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.632+0000 7fa8ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:19:16.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.632+0000 7fa8ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:16.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.632+0000 7fa8ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:16.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.632+0000 7fa8ae7fc640 10 monclient(hunting): my global_id is 4457 2024-03-25T18:19:16.633 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.632+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8b81442c0 con 0x7fa8b80b4670 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 688743204 0 0) 0x7fa8b81442c0 con 0x7fa8b80b4670 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8b81445a0 con 0x7fa8b80b4670 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa8b4001ca0 con 0x7fa8b80b4670 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:16.663 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3843765544 0 0) 0x7fa89c004eb0 con 0x7fa8b813c480 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8b81442c0 con 0x7fa8b813c480 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa8b0001b50 con 0x7fa8b813c480 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.662+0000 7fa8ae7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:16.665 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.663+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 683105979 0 0) 0x7fa8b8144070 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.663+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa89c004eb0 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.663+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa8a8001870 con 0x7fa8b80b4ff0 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.663+0000 7fa8ae7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.663+0000 7fa8ae7fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.663+0000 7fa8ae7fc640 10 monclient(hunting): dump: 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:16.666 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.667 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:16.667 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:16.667 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:16.667 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:16.667 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:16.667 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:16.667 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:16.680 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1808494553 0 0) 0x7fa8b81445a0 con 0x7fa8b80b4670 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 >> v1:172.21.15.183:6789/0 conn(0x7fa8b80b4ff0 legacy=0x7fa8b8130000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 >> v1:172.21.15.7:6789/0 conn(0x7fa8b813c480 legacy=0x7fa8b81e3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 1 monclient: found mon.c 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8b8143620 con 0x7fa8b80b4670 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 5 monclient: authenticate success, global_id 4457 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 10 monclient: _renew_subs 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8b8143b20 con 0x7fa8b80b4670 2024-03-25T18:19:16.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 10 client.4457.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 10 monclient: _renew_subs 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8b8145590 con 0x7fa8b80b4670 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.679+0000 7fa8c0123640 1 librados: init done 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.680+0000 7fa8c0123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.681+0000 7fa8c0123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.681+0000 7fa8c0123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:16.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:16.681+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8b81326c0 con 0x7fa8b80b4670 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.000+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8b40038a0 con 0x7fa8b80b4670 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.000+0000 7fa8ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.000+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fa8b4003b50 con 0x7fa8b80b4670 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.000+0000 7fa8ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.000+0000 7fa8ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.000+0000 7fa8ae7fc640 10 monclient: dump: 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:17.001 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:17.002 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:17.002 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:17.002 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:17.002 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:17.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.000+0000 7fa8affff640 4 set_mon_vals no callback set 2024-03-25T18:19:17.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.001+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fa8b4003d70 con 0x7fa8b80b4670 2024-03-25T18:19:17.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.001+0000 7fa8bd6be640 10 monclient: get_auth_request con 0x7fa89c06e0f0 auth_method 0 2024-03-25T18:19:17.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.001+0000 7fa8ae7fc640 10 client.4457.objecter ms_handle_connect 0x7fa89c06e0f0 2024-03-25T18:19:17.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fa8b4054660 con 0x7fa8b80b4670 2024-03-25T18:19:17.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 10 client.4457.objecter ms_dispatch 0x7fa8b8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:17.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 3 client.4457.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:17.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 3 client.4457.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:17.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 20 client.4457.objecter dump_active .. 0 homeless 2024-03-25T18:19:17.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa8b4050170 con 0x7fa8b80b4670 2024-03-25T18:19:17.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:17.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.010+0000 7fa8ae7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:17.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.270+0000 7fa8c0123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:17.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.270+0000 7fa8c0123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:17.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.270+0000 7fa8c0123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:17.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.270+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8b806dbd0 con 0x7fa8b80b4670 2024-03-25T18:19:17.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.303+0000 7fa8ae7fc640 1 -- 172.21.15.7:0/4250834290 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fa8b40520a0 con 0x7fa8b80b4670 2024-03-25T18:19:17.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.303+0000 7fa8ae7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:17.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.303+0000 7fa8ae7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:17.304 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:17.304 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:17.307 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 10 librados: watch_flush enter 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 10 librados: watch_flush exit 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 20 client.4457.objecter shutdown clearing up homeless session... 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 10 client.4457.objecter successfully canceled tick 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 >> v1:172.21.15.7:6800/1717564767 conn(0x7fa89c06e0f0 legacy=0x7fa89c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 10 monclient: shutdown 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 >> v1:172.21.15.7:6790/0 conn(0x7fa8b80b4670 legacy=0x7fa8b8144ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 shutdown_connections 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 >> 172.21.15.7:0/4250834290 conn(0x7fa8b80617a0 msgr2=0x7fa8b8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.306+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 shutdown_connections 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.307+0000 7fa8c0123640 1 -- 172.21.15.7:0/4250834290 wait complete. 2024-03-25T18:19:17.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:17.307+0000 7fa8c0123640 1 librados: shutdown 2024-03-25T18:19:17.319 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:24.127 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:24.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.246+0000 7f5b51a25640 10 monclient: get_monmap_and_config 2024-03-25T18:19:24.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.246+0000 7f5b51a25640 10 monclient: build_initial_monmap 2024-03-25T18:19:24.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.246+0000 7f5b51a25640 10 monclient: monmap: 2024-03-25T18:19:24.247 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:24.247 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:24.247 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:24.247347+0000 2024-03-25T18:19:24.247 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:24.247347+0000 2024-03-25T18:19:24.248 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:24.248 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:24.248 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:24.248 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:24.248 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:24.248 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.247+0000 7f5b51a25640 1 Processor -- start 2024-03-25T18:19:24.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.247+0000 7f5b51a25640 1 -- start start 2024-03-25T18:19:24.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.247+0000 7f5b51a25640 10 monclient: init 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 10 monclient(hunting): picked mon.noname-b con 0x7f5b4c06f460 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 10 monclient(hunting): picked mon.noname-c con 0x7f5b4c06f840 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 10 monclient(hunting): picked mon.noname-a con 0x7f5b4c0728f0 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4c071a40 con 0x7f5b4c0728f0 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4c071db0 con 0x7f5b4c06f840 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4c070650 con 0x7f5b4c06f460 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:24.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.248+0000 7f5b51a25640 10 monclient(hunting): authenticate will time out at 7606.352991s 2024-03-25T18:19:24.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.264+0000 7f5b4b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5b4c0728f0 0x7f5b4c14a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54634/0 (socket says 172.21.15.7:54634) 2024-03-25T18:19:24.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.264+0000 7f5b4b7fe640 1 -- 172.21.15.7:0/3568064953 learned_addr learned my addr 172.21.15.7:0/3568064953 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:24.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.271+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082795136 0 0) 0x7f5b4c071a40 con 0x7f5b4c0728f0 2024-03-25T18:19:24.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.271+0000 7f5b49ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.271+0000 7f5b49ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.271+0000 7f5b49ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.271+0000 7f5b49ffb640 10 monclient(hunting): my global_id is 4443 2024-03-25T18:19:24.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.271+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b34003520 con 0x7f5b4c0728f0 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.292+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2220161491 0 0) 0x7f5b34003520 con 0x7f5b4c0728f0 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.292+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 >> v1:172.21.15.7:6790/0 conn(0x7f5b4c06f460 legacy=0x7f5b4c142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 >> v1:172.21.15.183:6789/0 conn(0x7f5b4c06f840 legacy=0x7f5b4c0721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 1 monclient: found mon.noname-a 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4c070200 con 0x7f5b4c0728f0 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:24.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5b44001a10 con 0x7f5b4c0728f0 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b49ffb640 10 monclient: dump: 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:24.296 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b51a25640 5 monclient: authenticate success, global_id 4443 2024-03-25T18:19:24.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.293+0000 7f5b51a25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:24.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.297+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b44001b70 con 0x7f5b4c0728f0 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b49ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b49ffb640 1 -- 172.21.15.7:0/3568064953 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5b44001f00 con 0x7f5b4c0728f0 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b51a25640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b51a25640 4 set_mon_vals no callback set 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b49ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b49ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b49ffb640 10 monclient: dump: 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.299 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:24.300 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:24.300 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:24.300 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:24.300 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b51a25640 10 monclient: shutdown 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b51a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.298+0000 7f5b51a25640 1 -- 172.21.15.7:0/3568064953 >> v1:172.21.15.7:6789/0 conn(0x7f5b4c0728f0 legacy=0x7f5b4c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.299+0000 7f5b51a25640 1 -- 172.21.15.7:0/3568064953 shutdown_connections 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.299+0000 7f5b51a25640 1 -- 172.21.15.7:0/3568064953 >> 172.21.15.7:0/3568064953 conn(0x7f5b4c0b4bd0 msgr2=0x7f5b4c0b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.299+0000 7f5b51a25640 1 -- 172.21.15.7:0/3568064953 shutdown_connections 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.299+0000 7f5b51a25640 1 -- 172.21.15.7:0/3568064953 wait complete. 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.299+0000 7f5b51a25640 10 monclient: build_initial_monmap 2024-03-25T18:19:24.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.299+0000 7f5b51a25640 10 monclient: monmap: 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 1 librados: starting msgr at 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 1 librados: starting objecter 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 1 Processor -- start 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 1 -- start start 2024-03-25T18:19:24.303 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 1 librados: setting wanted keys 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 1 librados: calling monclient init 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 10 monclient: init 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.300+0000 7f5b51a25640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 10 monclient(hunting): picked mon.noname-a con 0x7f5b4c06f460 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 10 monclient(hunting): picked mon.noname-b con 0x7f5b4c06f840 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 10 monclient(hunting): picked mon.noname-c con 0x7f5b4c0728f0 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4c1d7890 con 0x7f5b4c06f460 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4c13ff00 con 0x7f5b4c06f840 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b4c135c30 con 0x7f5b4c0728f0 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:24.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.301+0000 7f5b51a25640 10 monclient(hunting): authenticate will time out at 7606.405763s 2024-03-25T18:19:24.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.322+0000 7f5b4b7fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f5b4c0728f0 0x7f5b4c135520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:44482/0 (socket says 172.21.15.7:44482) 2024-03-25T18:19:24.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.322+0000 7f5b4b7fe640 1 -- 172.21.15.7:0/4134942941 learned_addr learned my addr 172.21.15.7:0/4134942941 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:24.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.324+0000 7f5b33fff640 10 client.?.objecter ms_handle_connect 0x7f5b4c0728f0 2024-03-25T18:19:24.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.324+0000 7f5b33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:24.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.328+0000 7f5b33fff640 10 client.?.objecter ms_handle_connect 0x7f5b4c06f460 2024-03-25T18:19:24.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.328+0000 7f5b33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:24.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.328+0000 7f5b33fff640 10 client.?.objecter ms_handle_connect 0x7f5b4c06f840 2024-03-25T18:19:24.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.328+0000 7f5b33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:24.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.339+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2945606955 0 0) 0x7f5b4c13ff00 con 0x7f5b4c06f840 2024-03-25T18:19:24.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.339+0000 7f5b33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.339+0000 7f5b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.339+0000 7f5b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.339+0000 7f5b33fff640 10 monclient(hunting): my global_id is 4444 2024-03-25T18:19:24.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.339+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b38004eb0 con 0x7f5b4c06f840 2024-03-25T18:19:24.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.349+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2399823304 0 0) 0x7f5b4c135c30 con 0x7f5b4c0728f0 2024-03-25T18:19:24.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.349+0000 7f5b33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.349+0000 7f5b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.349+0000 7f5b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.349+0000 7f5b33fff640 10 monclient(hunting): my global_id is 4463 2024-03-25T18:19:24.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.349+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b4c13ff00 con 0x7f5b4c0728f0 2024-03-25T18:19:24.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.350+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227531831 0 0) 0x7f5b4c1d7890 con 0x7f5b4c06f460 2024-03-25T18:19:24.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.350+0000 7f5b33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:24.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.350+0000 7f5b33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:24.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.350+0000 7f5b33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:24.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.350+0000 7f5b33fff640 10 monclient(hunting): my global_id is 4446 2024-03-25T18:19:24.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.350+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b4c135c30 con 0x7f5b4c06f460 2024-03-25T18:19:24.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.362+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2541011210 0 0) 0x7f5b4c13ff00 con 0x7f5b4c0728f0 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.362+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b4c1d7890 con 0x7f5b4c0728f0 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.362+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5b44001e80 con 0x7f5b4c0728f0 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.362+0000 7f5b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.363+0000 7f5b33fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.363+0000 7f5b33fff640 10 monclient(hunting): dump: 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:24.364 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:24.365 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.363+0000 7f5b33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:24.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.374+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 173644568 0 0) 0x7f5b38004eb0 con 0x7f5b4c06f840 2024-03-25T18:19:24.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.374+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b4c13ff00 con 0x7f5b4c06f840 2024-03-25T18:19:24.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.374+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5b40001930 con 0x7f5b4c06f840 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.374+0000 7f5b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.374+0000 7f5b33fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.374+0000 7f5b33fff640 10 monclient(hunting): dump: 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:24.376 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.384+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3329215873 0 0) 0x7f5b4c135c30 con 0x7f5b4c06f460 2024-03-25T18:19:24.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.384+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5b38004eb0 con 0x7f5b4c06f460 2024-03-25T18:19:24.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.384+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5b3c001830 con 0x7f5b4c06f460 2024-03-25T18:19:24.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.384+0000 7f5b33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.384+0000 7f5b33fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.384+0000 7f5b33fff640 10 monclient(hunting): dump: 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:24.386 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2892953793 0 0) 0x7f5b4c1d7890 con 0x7f5b4c0728f0 2024-03-25T18:19:24.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 >> v1:172.21.15.183:6789/0 conn(0x7f5b4c06f840 legacy=0x7f5b4c134e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 >> v1:172.21.15.7:6789/0 conn(0x7f5b4c06f460 legacy=0x7f5b4c1d7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:24.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 1 monclient: found mon.c 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b4c1d8c90 con 0x7f5b4c0728f0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 5 monclient: authenticate success, global_id 4463 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 10 monclient: _renew_subs 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b4c1368f0 con 0x7f5b4c0728f0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 10 monclient: _renew_subs 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b4c0ae450 con 0x7f5b4c0728f0 2024-03-25T18:19:24.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.387+0000 7f5b51a25640 1 librados: init done 2024-03-25T18:19:24.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.388+0000 7f5b51a25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.388+0000 7f5b51a25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.388+0000 7f5b51a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:24.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.388+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b4c1d7a40 con 0x7f5b4c0728f0 2024-03-25T18:19:24.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.723+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5b44001fe0 con 0x7f5b4c0728f0 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.723+0000 7f5b33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.723+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5b44004e10 con 0x7f5b4c0728f0 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.723+0000 7f5b33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.723+0000 7f5b33fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.723+0000 7f5b33fff640 10 monclient: dump: 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:24.725 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:24.726 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:24.726 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:24.726 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:24.726 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:24.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.723+0000 7f5b497fa640 4 set_mon_vals no callback set 2024-03-25T18:19:24.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.724+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f5b4401ade0 con 0x7f5b4c0728f0 2024-03-25T18:19:24.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.725+0000 7f5b4affd640 10 monclient: get_auth_request con 0x7f5b3806e0f0 auth_method 0 2024-03-25T18:19:24.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.725+0000 7f5b33fff640 10 client.4463.objecter ms_handle_connect 0x7f5b3806e0f0 2024-03-25T18:19:24.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f5b44054940 con 0x7f5b4c0728f0 2024-03-25T18:19:24.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 10 client.4463.objecter ms_dispatch 0x7f5b4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:24.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:24.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:24.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-25T18:19:24.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5b44050450 con 0x7f5b4c0728f0 2024-03-25T18:19:24.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:24.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.730+0000 7f5b33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:24.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.955+0000 7f5b51a25640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.955+0000 7f5b51a25640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:24.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.955+0000 7f5b51a25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:24.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:24.955+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5b4c072d20 con 0x7f5b4c0728f0 2024-03-25T18:19:25.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.029+0000 7f5b33fff640 1 -- 172.21.15.7:0/4134942941 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f5b44052380 con 0x7f5b4c0728f0 2024-03-25T18:19:25.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.029+0000 7f5b33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:25.030 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.029+0000 7f5b33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:25.031 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:25.031 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.031+0000 7f5b51a25640 10 librados: watch_flush enter 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 10 librados: watch_flush exit 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 10 client.4463.objecter successfully canceled tick 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 >> v1:172.21.15.7:6800/1717564767 conn(0x7f5b3806e0f0 legacy=0x7f5b380705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 10 monclient: shutdown 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:25.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 >> v1:172.21.15.7:6790/0 conn(0x7f5b4c0728f0 legacy=0x7f5b4c135520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:25.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 shutdown_connections 2024-03-25T18:19:25.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 >> 172.21.15.7:0/4134942941 conn(0x7f5b4c0b4bd0 msgr2=0x7f5b4c0ae8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:25.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 shutdown_connections 2024-03-25T18:19:25.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 1 -- 172.21.15.7:0/4134942941 wait complete. 2024-03-25T18:19:25.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:25.032+0000 7f5b51a25640 1 librados: shutdown 2024-03-25T18:19:25.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:31.851 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.971+0000 7fd08c886640 10 monclient: get_monmap_and_config 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.971+0000 7fd08c886640 10 monclient: build_initial_monmap 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.971+0000 7fd08c886640 10 monclient: monmap: 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:31.972527+0000 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:31.972527+0000 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:31.972 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:31.973 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:31.973 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:31.973 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:31.973 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.972+0000 7fd08c886640 1 Processor -- start 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.972+0000 7fd08c886640 1 -- start start 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.972+0000 7fd08c886640 10 monclient: init 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 10 monclient(hunting): picked mon.noname-c con 0x7fd0880b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 10 monclient(hunting): picked mon.noname-a con 0x7fd0880b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 10 monclient(hunting): picked mon.noname-b con 0x7fd08813c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880ab0e0 con 0x7fd0880b4670 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880ab3c0 con 0x7fd0880b4ff0 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd0880a9f30 con 0x7fd08813c480 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:31.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.973+0000 7fd08c886640 10 monclient(hunting): authenticate will time out at 7614.078038s 2024-03-25T18:19:31.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.979+0000 7fd08659c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd0880b4ff0 0x7fd08813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:37956/0 (socket says 172.21.15.7:37956) 2024-03-25T18:19:31.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.979+0000 7fd08659c640 1 -- 172.21.15.7:0/2530755134 learned_addr learned my addr 172.21.15.7:0/2530755134 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:31.994 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.993+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3630757042 0 0) 0x7fd0880ab3c0 con 0x7fd0880b4ff0 2024-03-25T18:19:31.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.993+0000 7fd08559a640 10 monclient(hunting): _negotiate 2024-03-25T18:19:31.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.993+0000 7fd08559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:31.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.993+0000 7fd08559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:31.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.993+0000 7fd08559a640 10 monclient(hunting): my global_id is 4447 2024-03-25T18:19:31.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.994+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd070003520 con 0x7fd0880b4ff0 2024-03-25T18:19:31.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 495454132 0 0) 0x7fd070003520 con 0x7fd0880b4ff0 2024-03-25T18:19:31.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 >> v1:172.21.15.7:6790/0 conn(0x7fd08813c480 legacy=0x7fd08814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:31.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 >> v1:172.21.15.7:6789/0 conn(0x7fd0880b4670 legacy=0x7fd0881425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:19:31.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 10 monclient: _finish_hunting 0 2024-03-25T18:19:31.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 1 monclient: found mon.noname-c 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0880a9b70 con 0x7fd0880b4ff0 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 10 monclient: _finish_auth 0 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 10 monclient: _check_auth_tickets 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd078001690 con 0x7fd0880b4ff0 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08559a640 10 monclient: dump: 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.000 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:32.001 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:32.001 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:32.001 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:32.001 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:32.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08c886640 5 monclient: authenticate success, global_id 4447 2024-03-25T18:19:32.001 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:31.998+0000 7fd08c886640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:32.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd078001860 con 0x7fd0880b4ff0 2024-03-25T18:19:32.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:32.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08559a640 1 -- 172.21.15.7:0/2530755134 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd078001cb0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:32.012 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08559a640 10 monclient: dump: 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08c886640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:32.013 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.011+0000 7fd08c886640 4 set_mon_vals no callback set 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.012+0000 7fd08c886640 10 monclient: shutdown 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.012+0000 7fd08c886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.012+0000 7fd08c886640 1 -- 172.21.15.7:0/2530755134 >> v1:172.21.15.183:6789/0 conn(0x7fd0880b4ff0 legacy=0x7fd08813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.012+0000 7fd08c886640 1 -- 172.21.15.7:0/2530755134 shutdown_connections 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.012+0000 7fd08c886640 1 -- 172.21.15.7:0/2530755134 >> 172.21.15.7:0/2530755134 conn(0x7fd0880617a0 msgr2=0x7fd088061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.012+0000 7fd08c886640 1 -- 172.21.15.7:0/2530755134 shutdown_connections 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.012+0000 7fd08c886640 1 -- 172.21.15.7:0/2530755134 wait complete. 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 10 monclient: build_initial_monmap 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 10 monclient: monmap: 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:32.016 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 1 librados: starting msgr at 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 1 librados: starting objecter 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 1 Processor -- start 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 1 -- start start 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 1 librados: setting wanted keys 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 1 librados: calling monclient init 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.013+0000 7fd08c886640 10 monclient: init 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:32.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 10 monclient(hunting): picked mon.noname-a con 0x7fd0880b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 10 monclient(hunting): picked mon.noname-c con 0x7fd0880b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 10 monclient(hunting): picked mon.noname-b con 0x7fd08813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd088144700 con 0x7fd0880b4670 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd088130cc0 con 0x7fd08813c480 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd088144970 con 0x7fd0880b4ff0 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:32.018 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.014+0000 7fd08c886640 10 monclient(hunting): authenticate will time out at 7614.119242s 2024-03-25T18:19:32.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.021+0000 7fd086d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd08813c480 0x7fd0881e2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:37968/0 (socket says 172.21.15.7:37968) 2024-03-25T18:19:32.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.021+0000 7fd086d9d640 1 -- 172.21.15.7:0/2338018401 learned_addr learned my addr 172.21.15.7:0/2338018401 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:32.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.023+0000 7fd076ffd640 10 client.?.objecter ms_handle_connect 0x7fd0880b4ff0 2024-03-25T18:19:32.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.023+0000 7fd076ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:32.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.026+0000 7fd076ffd640 10 client.?.objecter ms_handle_connect 0x7fd08813c480 2024-03-25T18:19:32.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.026+0000 7fd076ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:32.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.033+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045900074 0 0) 0x7fd088144970 con 0x7fd0880b4ff0 2024-03-25T18:19:32.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.033+0000 7fd076ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:32.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.033+0000 7fd076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:32.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.033+0000 7fd076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:32.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.033+0000 7fd076ffd640 10 monclient(hunting): my global_id is 4466 2024-03-25T18:19:32.034 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.033+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd068004eb0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.039+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1836130506 0 0) 0x7fd088130cc0 con 0x7fd08813c480 2024-03-25T18:19:32.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.039+0000 7fd076ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:32.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.039+0000 7fd076ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:32.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.039+0000 7fd076ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:32.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.039+0000 7fd076ffd640 10 monclient(hunting): my global_id is 4450 2024-03-25T18:19:32.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.039+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd088144970 con 0x7fd08813c480 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.045+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 335331315 0 0) 0x7fd068004eb0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.045+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd088130cc0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.045+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd07c001510 con 0x7fd0880b4ff0 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.045+0000 7fd076ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.045+0000 7fd076ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.045+0000 7fd076ffd640 10 monclient(hunting): dump: 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:32.046 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:32.047 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:32.047 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:32.047 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:32.047 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.045+0000 7fd076ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.050+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3239285009 0 0) 0x7fd088130cc0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 >> v1:172.21.15.183:6789/0 conn(0x7fd08813c480 legacy=0x7fd0881e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 >> v1:172.21.15.7:6789/0 conn(0x7fd0880b4670 legacy=0x7fd0881459a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 1 monclient: found mon.c 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:32.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd088144c50 con 0x7fd0880b4ff0 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd076ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 5 monclient: authenticate success, global_id 4466 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 10 monclient: _renew_subs 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd088144f30 con 0x7fd0880b4ff0 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 10 client.4466.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 10 monclient: _renew_subs 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd08813dc70 con 0x7fd0880b4ff0 2024-03-25T18:19:32.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.051+0000 7fd08c886640 1 librados: init done 2024-03-25T18:19:32.054 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.052+0000 7fd08c886640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:32.054 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.052+0000 7fd08c886640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:32.054 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.052+0000 7fd08c886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:32.054 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.052+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd088143670 con 0x7fd0880b4ff0 2024-03-25T18:19:32.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.366+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd07c001a20 con 0x7fd0880b4ff0 2024-03-25T18:19:32.367 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.366+0000 7fd076ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.366+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd07c003910 con 0x7fd0880b4ff0 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.366+0000 7fd076ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.366+0000 7fd076ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.366+0000 7fd076ffd640 10 monclient: dump: 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:32.368 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:32.369 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:32.369 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.366+0000 7fd084d99640 4 set_mon_vals no callback set 2024-03-25T18:19:32.369 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.367+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fd07c0198e0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.369 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.368+0000 7fd08659c640 10 monclient: get_auth_request con 0x7fd06806dce0 auth_method 0 2024-03-25T18:19:32.369 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.368+0000 7fd076ffd640 10 client.4466.objecter ms_handle_connect 0x7fd06806dce0 2024-03-25T18:19:32.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fd07c0535a0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 10 client.4466.objecter ms_dispatch 0x7fd088000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:32.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 3 client.4466.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:32.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 3 client.4466.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:32.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 20 client.4466.objecter dump_active .. 0 homeless 2024-03-25T18:19:32.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd07c04f0b0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:32.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.374+0000 7fd076ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:32.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.604+0000 7fd08c886640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.604+0000 7fd08c886640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.604+0000 7fd08c886640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:32.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.604+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd08806dbd0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.641 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.641+0000 7fd076ffd640 1 -- 172.21.15.7:0/2338018401 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fd07c050fe0 con 0x7fd0880b4ff0 2024-03-25T18:19:32.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.641+0000 7fd076ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:32.642 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.641+0000 7fd076ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:32.642 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:32.642 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:32.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.642+0000 7fd08c886640 10 librados: watch_flush enter 2024-03-25T18:19:32.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.642+0000 7fd08c886640 10 librados: watch_flush exit 2024-03-25T18:19:32.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 20 client.4466.objecter shutdown clearing up homeless session... 2024-03-25T18:19:32.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 10 client.4466.objecter successfully canceled tick 2024-03-25T18:19:32.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 >> v1:172.21.15.7:6800/1717564767 conn(0x7fd06806dce0 legacy=0x7fd0680701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 10 monclient: shutdown 2024-03-25T18:19:32.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:32.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 >> v1:172.21.15.7:6790/0 conn(0x7fd0880b4ff0 legacy=0x7fd0881460b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:32.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 shutdown_connections 2024-03-25T18:19:32.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 >> 172.21.15.7:0/2338018401 conn(0x7fd0880617a0 msgr2=0x7fd088140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:32.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 shutdown_connections 2024-03-25T18:19:32.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 1 -- 172.21.15.7:0/2338018401 wait complete. 2024-03-25T18:19:32.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:32.643+0000 7fd08c886640 1 librados: shutdown 2024-03-25T18:19:32.654 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:33.970 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:40.607 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.026 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:40.663 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.062 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:05:40.699 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.075 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:40.701 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.079 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:40.706 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.090 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:40.727 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.096 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:40.733 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.113 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:05:40.739 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:34.123 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:05:40.760 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:39.461 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:39.575 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.574+0000 7f830e6ef640 10 monclient: get_monmap_and_config 2024-03-25T18:19:39.575 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.574+0000 7f830e6ef640 10 monclient: build_initial_monmap 2024-03-25T18:19:39.575 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.574+0000 7f830e6ef640 10 monclient: monmap: 2024-03-25T18:19:39.575 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:39.575 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:39.575 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:39.575621+0000 2024-03-25T18:19:39.576 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:39.575621+0000 2024-03-25T18:19:39.576 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:39.576 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:39.576 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:39.576 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:39.576 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:39.576 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.575+0000 7f830e6ef640 1 Processor -- start 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.575+0000 7f830e6ef640 1 -- start start 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.575+0000 7f830e6ef640 10 monclient: init 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.575+0000 7f830e6ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 10 monclient(hunting): picked mon.noname-a con 0x7f83080b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 10 monclient(hunting): picked mon.noname-b con 0x7f83080b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 10 monclient(hunting): picked mon.noname-c con 0x7f830813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:39.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080ab0e0 con 0x7f83080b4ff0 2024-03-25T18:19:39.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080ab3c0 con 0x7f830813c480 2024-03-25T18:19:39.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080a9f30 con 0x7f83080b4670 2024-03-25T18:19:39.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:39.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.576+0000 7f830e6ef640 10 monclient(hunting): authenticate will time out at 7621.680740s 2024-03-25T18:19:39.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.577+0000 7f830cc8c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f830813c480 0x7f830814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:42310/0 (socket says 172.21.15.7:42310) 2024-03-25T18:19:39.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.577+0000 7f830cc8c640 1 -- 172.21.15.7:0/804347197 learned_addr learned my addr 172.21.15.7:0/804347197 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:39.681 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.680+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 318866489 0 0) 0x7f83080ab0e0 con 0x7f83080b4ff0 2024-03-25T18:19:39.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.681+0000 7f8306ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.681+0000 7f8306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.681+0000 7f8306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.681+0000 7f8306ffd640 10 monclient(hunting): my global_id is 4449 2024-03-25T18:19:39.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.681+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f0003520 con 0x7f83080b4ff0 2024-03-25T18:19:39.699 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.699+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4099969520 0 0) 0x7f83080a9f30 con 0x7f83080b4670 2024-03-25T18:19:39.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.699+0000 7f8306ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.699+0000 7f8306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.699+0000 7f8306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.699+0000 7f8306ffd640 10 monclient(hunting): my global_id is 4469 2024-03-25T18:19:39.700 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.699+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83080ab0e0 con 0x7f83080b4670 2024-03-25T18:19:39.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.701+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 739810545 0 0) 0x7f83080ab3c0 con 0x7f830813c480 2024-03-25T18:19:39.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.701+0000 7f8306ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.701+0000 7f8306ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.701+0000 7f8306ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.701+0000 7f8306ffd640 10 monclient(hunting): my global_id is 4453 2024-03-25T18:19:39.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.701+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83080a9f30 con 0x7f830813c480 2024-03-25T18:19:39.704 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2802019769 0 0) 0x7f82f0003520 con 0x7f83080b4ff0 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 >> v1:172.21.15.7:6790/0 conn(0x7f83080b4670 legacy=0x7f83081425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 >> v1:172.21.15.183:6789/0 conn(0x7f830813c480 legacy=0x7f830814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 1 monclient: found mon.noname-a 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83080a9b70 con 0x7f83080b4ff0 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 10 monclient: _finish_auth 0 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f82f8001690 con 0x7f83080b4ff0 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f830e6ef640 5 monclient: authenticate success, global_id 4449 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f830e6ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:39.705 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f8306ffd640 10 monclient: dump: 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:39.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.703+0000 7f830e6ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:39.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82f80017f0 con 0x7f83080b4ff0 2024-03-25T18:19:39.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f8306ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f8306ffd640 1 -- 172.21.15.7:0/804347197 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f82f8001c00 con 0x7f83080b4ff0 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f830e6ef640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f830e6ef640 4 set_mon_vals no callback set 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f8306ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f8306ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.714+0000 7f8306ffd640 10 monclient: dump: 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:39.716 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:39.717 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:39.717 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.715+0000 7f830e6ef640 10 monclient: shutdown 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.715+0000 7f830e6ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.715+0000 7f830e6ef640 1 -- 172.21.15.7:0/804347197 >> v1:172.21.15.7:6789/0 conn(0x7f83080b4ff0 legacy=0x7f830813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.715+0000 7f830e6ef640 1 -- 172.21.15.7:0/804347197 shutdown_connections 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.715+0000 7f830e6ef640 1 -- 172.21.15.7:0/804347197 >> 172.21.15.7:0/804347197 conn(0x7f83080617a0 msgr2=0x7f8308061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.715+0000 7f830e6ef640 1 -- 172.21.15.7:0/804347197 shutdown_connections 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.715+0000 7f830e6ef640 1 -- 172.21.15.7:0/804347197 wait complete. 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.716+0000 7f830e6ef640 10 monclient: build_initial_monmap 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.716+0000 7f830e6ef640 10 monclient: monmap: 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:39.719 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.716+0000 7f830e6ef640 1 librados: starting msgr at 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.716+0000 7f830e6ef640 1 librados: starting objecter 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.716+0000 7f830e6ef640 1 Processor -- start 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 1 -- start start 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 1 librados: setting wanted keys 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 1 librados: calling monclient init 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient: init 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:39.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient(hunting): picked mon.noname-b con 0x7f83080b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient(hunting): picked mon.noname-c con 0x7f83080b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient(hunting): picked mon.noname-a con 0x7f830813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8308135ea0 con 0x7f830813c480 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080ac3d0 con 0x7f83080b4670 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83080ac6b0 con 0x7f83080b4ff0 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:39.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.717+0000 7f830e6ef640 10 monclient(hunting): authenticate will time out at 7621.822364s 2024-03-25T18:19:39.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.722+0000 7f8307fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f83080b4670 0x7f830813b8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:42352/0 (socket says 172.21.15.7:42352) 2024-03-25T18:19:39.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.722+0000 7f8307fff640 1 -- 172.21.15.7:0/1637802290 learned_addr learned my addr 172.21.15.7:0/1637802290 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:39.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.724+0000 7f8304ff9640 10 client.?.objecter ms_handle_connect 0x7f83080b4670 2024-03-25T18:19:39.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.724+0000 7f8304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.729+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1632389196 0 0) 0x7f83080ac3d0 con 0x7f83080b4670 2024-03-25T18:19:39.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.729+0000 7f8304ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.730 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.729+0000 7f8304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.729+0000 7f8304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.729+0000 7f8304ff9640 10 monclient(hunting): my global_id is 4456 2024-03-25T18:19:39.731 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.729+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f4004eb0 con 0x7f83080b4670 2024-03-25T18:19:39.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.733+0000 7f8304ff9640 10 client.?.objecter ms_handle_connect 0x7f83080b4ff0 2024-03-25T18:19:39.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.733+0000 7f8304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.734+0000 7f8304ff9640 10 client.?.objecter ms_handle_connect 0x7f830813c480 2024-03-25T18:19:39.735 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.734+0000 7f8304ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:39.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.745+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 319018640 0 0) 0x7f8308135ea0 con 0x7f830813c480 2024-03-25T18:19:39.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.745+0000 7f8304ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.745+0000 7f8304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.745+0000 7f8304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.745+0000 7f8304ff9640 10 monclient(hunting): my global_id is 4452 2024-03-25T18:19:39.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.745+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83080ac3d0 con 0x7f830813c480 2024-03-25T18:19:39.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.747+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3650031233 0 0) 0x7f82f4004eb0 con 0x7f83080b4670 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.747+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8308135ea0 con 0x7f83080b4670 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.747+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f82f8001bc0 con 0x7f83080b4670 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.747+0000 7f8304ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.747+0000 7f8304ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.747+0000 7f8304ff9640 10 monclient(hunting): dump: 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:39.749 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:39.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.747+0000 7f8304ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:39.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.758+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3829818285 0 0) 0x7f83080ac6b0 con 0x7f83080b4ff0 2024-03-25T18:19:39.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.758+0000 7f8304ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:19:39.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.758+0000 7f8304ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:39.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.758+0000 7f8304ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:39.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.758+0000 7f8304ff9640 10 monclient(hunting): my global_id is 4472 2024-03-25T18:19:39.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.758+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f82f4004eb0 con 0x7f83080b4ff0 2024-03-25T18:19:39.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3104937719 0 0) 0x7f8308135ea0 con 0x7f83080b4670 2024-03-25T18:19:39.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 >> v1:172.21.15.7:6790/0 conn(0x7f83080b4ff0 legacy=0x7f83080b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 >> v1:172.21.15.7:6789/0 conn(0x7f830813c480 legacy=0x7f83081362d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 1 monclient: found mon.b 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830813c110 con 0x7f83080b4670 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 10 monclient: _finish_auth 0 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f8304ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 5 monclient: authenticate success, global_id 4456 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 10 monclient: _renew_subs 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f830813ae10 con 0x7f83080b4670 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 10 monclient: _renew_subs 2024-03-25T18:19:39.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:39.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83080b1550 con 0x7f83080b4670 2024-03-25T18:19:39.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.759+0000 7f830e6ef640 1 librados: init done 2024-03-25T18:19:39.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.760+0000 7f830e6ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.760+0000 7f830e6ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:39.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.760+0000 7f830e6ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:39.762 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:39.760+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f830806dbd0 con 0x7f83080b4670 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.030+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f82f80011f0 con 0x7f83080b4670 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.030+0000 7f8304ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.030+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f82f8003b00 con 0x7f83080b4670 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.030+0000 7f8304ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.030+0000 7f8305ffb640 4 set_mon_vals no callback set 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.030+0000 7f8304ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.030+0000 7f8304ff9640 10 monclient: dump: 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:40.031 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:40.032 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:40.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.031+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f82f8003ce0 con 0x7f83080b4670 2024-03-25T18:19:40.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.032+0000 7f83077fe640 10 monclient: get_auth_request con 0x7f82f406de00 auth_method 0 2024-03-25T18:19:40.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.032+0000 7f8304ff9640 10 client.4456.objecter ms_handle_connect 0x7f82f406de00 2024-03-25T18:19:40.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.036+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f82f8054950 con 0x7f83080b4670 2024-03-25T18:19:40.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.036+0000 7f8304ff9640 10 client.4456.objecter ms_dispatch 0x7f8308000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:40.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.036+0000 7f8304ff9640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:40.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.036+0000 7f8304ff9640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:40.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.036+0000 7f8304ff9640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-25T18:19:40.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.037+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f82f8050290 con 0x7f83080b4670 2024-03-25T18:19:40.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.037+0000 7f8304ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:40.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.037+0000 7f8304ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:40.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.285+0000 7f830e6ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:40.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.285+0000 7f830e6ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:40.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.285+0000 7f830e6ef640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:40.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.285+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8308146790 con 0x7f83080b4670 2024-03-25T18:19:40.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.317+0000 7f8304ff9640 1 -- 172.21.15.7:0/1637802290 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f82f8052390 con 0x7f83080b4670 2024-03-25T18:19:40.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.317+0000 7f8304ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:40.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.317+0000 7f8304ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:40.319 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:40.319 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 10 librados: watch_flush enter 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 10 librados: watch_flush exit 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 10 client.4456.objecter successfully canceled tick 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 >> v1:172.21.15.7:6800/1717564767 conn(0x7f82f406de00 legacy=0x7f82f40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 10 monclient: shutdown 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 >> v1:172.21.15.183:6789/0 conn(0x7f83080b4670 legacy=0x7f830813b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 shutdown_connections 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 >> 172.21.15.7:0/1637802290 conn(0x7f83080617a0 msgr2=0x7f83081407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 shutdown_connections 2024-03-25T18:19:40.322 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 1 -- 172.21.15.7:0/1637802290 wait complete. 2024-03-25T18:19:40.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:40.320+0000 7f830e6ef640 1 librados: shutdown 2024-03-25T18:19:40.333 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:47.139 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.255+0000 7f659f16b640 10 monclient: get_monmap_and_config 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.255+0000 7f659f16b640 10 monclient: build_initial_monmap 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.255+0000 7f659f16b640 10 monclient: monmap: 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:47.256963+0000 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:47.256963+0000 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:47.257 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.256+0000 7f659f16b640 1 Processor -- start 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.256+0000 7f659f16b640 1 -- start start 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.256+0000 7f659f16b640 10 monclient: init 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 10 monclient(hunting): picked mon.noname-c con 0x7f65980b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 10 monclient(hunting): picked mon.noname-b con 0x7f65980b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 10 monclient(hunting): picked mon.noname-a con 0x7f65980b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980a96f0 con 0x7f65980b1b00 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65980ab0e0 con 0x7f65980b4e70 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6598145f70 con 0x7f65980b44b0 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:47.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.257+0000 7f659f16b640 10 monclient(hunting): authenticate will time out at 7629.362316s 2024-03-25T18:19:47.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.259+0000 7f6597fff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f65980b44b0 0x7f6598145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:60242/0 (socket says 172.21.15.7:60242) 2024-03-25T18:19:47.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.259+0000 7f6597fff640 1 -- 172.21.15.7:0/4247144442 learned_addr learned my addr 172.21.15.7:0/4247144442 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:47.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2007037079 0 0) 0x7f65980ab0e0 con 0x7f65980b4e70 2024-03-25T18:19:47.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): my global_id is 4459 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6588003520 con 0x7f65980b4e70 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899756229 0 0) 0x7f6598145f70 con 0x7f65980b44b0 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 10 monclient(hunting): my global_id is 4475 2024-03-25T18:19:47.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.276+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65980ab0e0 con 0x7f65980b44b0 2024-03-25T18:19:47.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1032403108 0 0) 0x7f6588003520 con 0x7f65980b4e70 2024-03-25T18:19:47.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 >> v1:172.21.15.7:6790/0 conn(0x7f65980b44b0 legacy=0x7f6598145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 >> v1:172.21.15.7:6789/0 conn(0x7f65980b1b00 legacy=0x7f659814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 10 monclient: _finish_hunting 0 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 1 monclient: found mon.noname-c 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65980a9b80 con 0x7f65980b4e70 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 10 monclient: _finish_auth 0 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 10 monclient: _check_auth_tickets 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f658c001690 con 0x7f65980b4e70 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f65977fe640 10 monclient: dump: 2024-03-25T18:19:47.287 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f659f16b640 5 monclient: authenticate success, global_id 4459 2024-03-25T18:19:47.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.285+0000 7f659f16b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:47.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f658c0017f0 con 0x7f65980b4e70 2024-03-25T18:19:47.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f65977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:47.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f65977fe640 1 -- 172.21.15.7:0/4247144442 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f658c001c00 con 0x7f65980b4e70 2024-03-25T18:19:47.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f659f16b640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f659f16b640 4 set_mon_vals no callback set 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f65977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f65977fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.293+0000 7f65977fe640 10 monclient: dump: 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:47.295 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:47.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.294+0000 7f659f16b640 10 monclient: shutdown 2024-03-25T18:19:47.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.294+0000 7f659f16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.294+0000 7f659f16b640 1 -- 172.21.15.7:0/4247144442 >> v1:172.21.15.183:6789/0 conn(0x7f65980b4e70 legacy=0x7f659813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.294+0000 7f659f16b640 1 -- 172.21.15.7:0/4247144442 shutdown_connections 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.294+0000 7f659f16b640 1 -- 172.21.15.7:0/4247144442 >> 172.21.15.7:0/4247144442 conn(0x7f65980617a0 msgr2=0x7f6598061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.294+0000 7f659f16b640 1 -- 172.21.15.7:0/4247144442 shutdown_connections 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.294+0000 7f659f16b640 1 -- 172.21.15.7:0/4247144442 wait complete. 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.295+0000 7f659f16b640 10 monclient: build_initial_monmap 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.295+0000 7f659f16b640 10 monclient: monmap: 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:47.298 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.295+0000 7f659f16b640 1 librados: starting msgr at 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.295+0000 7f659f16b640 1 librados: starting objecter 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.295+0000 7f659f16b640 1 Processor -- start 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 1 -- start start 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 1 librados: setting wanted keys 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 1 librados: calling monclient init 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient: init 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient(hunting): picked mon.noname-b con 0x7f65980b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient(hunting): picked mon.noname-a con 0x7f65980b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient(hunting): picked mon.noname-c con 0x7f65980b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:47.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6598143530 con 0x7f65980b44b0 2024-03-25T18:19:47.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6598131b80 con 0x7f65980b1b00 2024-03-25T18:19:47.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6598131e60 con 0x7f65980b4e70 2024-03-25T18:19:47.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:47.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.296+0000 7f659f16b640 10 monclient(hunting): authenticate will time out at 7629.401396s 2024-03-25T18:19:47.305 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.304+0000 7f659d708640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f65980b4e70 0x7f6598132c70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:60294/0 (socket says 172.21.15.7:60294) 2024-03-25T18:19:47.305 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.304+0000 7f659d708640 1 -- 172.21.15.7:0/3042059757 learned_addr learned my addr 172.21.15.7:0/3042059757 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:47.306 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.305+0000 7f65957fa640 10 client.?.objecter ms_handle_connect 0x7f65980b4e70 2024-03-25T18:19:47.306 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.305+0000 7f65957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.310+0000 7f65957fa640 10 client.?.objecter ms_handle_connect 0x7f65980b1b00 2024-03-25T18:19:47.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.310+0000 7f65957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.317+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 856933345 0 0) 0x7f6598131b80 con 0x7f65980b1b00 2024-03-25T18:19:47.318 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.317+0000 7f65957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.317+0000 7f65957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.317+0000 7f65957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.317+0000 7f65957fa640 10 monclient(hunting): my global_id is 4462 2024-03-25T18:19:47.319 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.317+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f657c004eb0 con 0x7f65980b1b00 2024-03-25T18:19:47.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.324+0000 7f65957fa640 10 client.?.objecter ms_handle_connect 0x7f65980b44b0 2024-03-25T18:19:47.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.324+0000 7f65957fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.332+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 595174273 0 0) 0x7f657c004eb0 con 0x7f65980b1b00 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.332+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6598131b80 con 0x7f65980b1b00 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.332+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f658c001bc0 con 0x7f65980b1b00 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.332+0000 7f65957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.332+0000 7f65957fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.332+0000 7f65957fa640 10 monclient(hunting): dump: 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.334 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.335 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:47.335 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:47.335 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:47.335 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:47.335 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:47.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.332+0000 7f65957fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:47.341 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.340+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2298843480 0 0) 0x7f6598131e60 con 0x7f65980b4e70 2024-03-25T18:19:47.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.340+0000 7f65957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.340+0000 7f65957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.340+0000 7f65957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.340+0000 7f65957fa640 10 monclient(hunting): my global_id is 4478 2024-03-25T18:19:47.342 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.340+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f657c004eb0 con 0x7f65980b4e70 2024-03-25T18:19:47.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.342+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2998786992 0 0) 0x7f6598143530 con 0x7f65980b44b0 2024-03-25T18:19:47.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.342+0000 7f65957fa640 10 monclient(hunting): _negotiate 2024-03-25T18:19:47.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.342+0000 7f65957fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:47.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.342+0000 7f65957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:47.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.342+0000 7f65957fa640 10 monclient(hunting): my global_id is 4458 2024-03-25T18:19:47.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.342+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6598131e60 con 0x7f65980b44b0 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 45509425 0 0) 0x7f6598131b80 con 0x7f65980b1b00 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 >> v1:172.21.15.7:6790/0 conn(0x7f65980b4e70 legacy=0x7f6598132c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 >> v1:172.21.15.7:6789/0 conn(0x7f65980b44b0 legacy=0x7f6598132560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 10 monclient: _finish_hunting 0 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 1 monclient: found mon.b 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6598132140 con 0x7f65980b1b00 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 10 monclient: _finish_auth 0 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 10 monclient: _check_auth_tickets 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f65957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f659f16b640 5 monclient: authenticate success, global_id 4462 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f659f16b640 10 monclient: _renew_subs 2024-03-25T18:19:47.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f659f16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.347+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65981425b0 con 0x7f65980b1b00 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 10 client.4462.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 10 monclient: _renew_subs 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6598133560 con 0x7f65980b1b00 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 1 librados: init done 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:47.351 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.348+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f659804f730 con 0x7f65980b1b00 2024-03-25T18:19:47.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f658c0038a0 con 0x7f65980b1b00 2024-03-25T18:19:47.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f65957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:47.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f658c003a10 con 0x7f65980b1b00 2024-03-25T18:19:47.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f65957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:47.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f65957fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:47.638 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f65957fa640 10 monclient: dump: 2024-03-25T18:19:47.638 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:47.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f6596ffd640 4 set_mon_vals no callback set 2024-03-25T18:19:47.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.637+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f658c0199e0 con 0x7f65980b1b00 2024-03-25T18:19:47.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.638+0000 7f6597fff640 10 monclient: get_auth_request con 0x7f657c06de80 auth_method 0 2024-03-25T18:19:47.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.638+0000 7f65957fa640 10 client.4462.objecter ms_handle_connect 0x7f657c06de80 2024-03-25T18:19:47.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.644+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f658c0534d0 con 0x7f65980b1b00 2024-03-25T18:19:47.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.644+0000 7f65957fa640 10 client.4462.objecter ms_dispatch 0x7f6598000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:47.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.644+0000 7f65957fa640 3 client.4462.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:47.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.644+0000 7f65957fa640 3 client.4462.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:47.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.644+0000 7f65957fa640 20 client.4462.objecter dump_active .. 0 homeless 2024-03-25T18:19:47.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.647+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f658c04efe0 con 0x7f65980b1b00 2024-03-25T18:19:47.648 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.647+0000 7f65957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:47.648 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.647+0000 7f65957fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:47.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.877+0000 7f659f16b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.877+0000 7f659f16b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.877+0000 7f659f16b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:19:47.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.877+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65980aa090 con 0x7f65980b1b00 2024-03-25T18:19:47.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.901+0000 7f65957fa640 1 -- 172.21.15.7:0/3042059757 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f658c050f10 con 0x7f65980b1b00 2024-03-25T18:19:47.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.901+0000 7f65957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.901+0000 7f65957fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:47.903 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 10 librados: watch_flush enter 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 10 librados: watch_flush exit 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 20 client.4462.objecter shutdown clearing up homeless session... 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 10 client.4462.objecter successfully canceled tick 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 >> v1:172.21.15.7:6800/1717564767 conn(0x7f657c06de80 legacy=0x7f657c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 10 monclient: shutdown 2024-03-25T18:19:47.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:47.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 >> v1:172.21.15.183:6789/0 conn(0x7f65980b1b00 legacy=0x7f6598141ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:47.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 shutdown_connections 2024-03-25T18:19:47.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.903+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 >> 172.21.15.7:0/3042059757 conn(0x7f65980617a0 msgr2=0x7f65980b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:47.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.904+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 shutdown_connections 2024-03-25T18:19:47.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.904+0000 7f659f16b640 1 -- 172.21.15.7:0/3042059757 wait complete. 2024-03-25T18:19:47.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:47.904+0000 7f659f16b640 1 librados: shutdown 2024-03-25T18:19:47.916 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:19:54.076 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:00.713 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.081 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:00.718 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.111 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:00.748 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.122 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:00.748 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.129 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:00.755 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.137 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:00.775 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.146 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:00.784 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.155 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:00.782 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.196 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:00.834 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:19:54.723 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:19:54.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.836+0000 7fd1e7888640 10 monclient: get_monmap_and_config 2024-03-25T18:19:54.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.836+0000 7fd1e7888640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.836+0000 7fd1e7888640 10 monclient: monmap: 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:19:54.837745+0000 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:19:54.837745+0000 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:19:54.838 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:54.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.837+0000 7fd1e7888640 1 Processor -- start 2024-03-25T18:19:54.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.837+0000 7fd1e7888640 1 -- start start 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.837+0000 7fd1e7888640 10 monclient: init 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e00b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e00b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00ab0e0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00ab3c0 con 0x7fd1e013c480 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e00a9f30 con 0x7fd1e00b4670 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.838+0000 7fd1e7888640 10 monclient(hunting): authenticate will time out at 7636.943197s 2024-03-25T18:19:54.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.845+0000 7fd1e5624640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fd1e00b4ff0 0x7fd1e013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:41004/0 (socket says 172.21.15.7:41004) 2024-03-25T18:19:54.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.845+0000 7fd1e5624640 1 -- 172.21.15.7:0/666897150 learned_addr learned my addr 172.21.15.7:0/666897150 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:54.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.857+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1607811183 0 0) 0x7fd1e00ab3c0 con 0x7fd1e013c480 2024-03-25T18:19:54.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.857+0000 7fd1d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.857+0000 7fd1d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.857+0000 7fd1d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.857+0000 7fd1d7fff640 10 monclient(hunting): my global_id is 4465 2024-03-25T18:19:54.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.857+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1cc003520 con 0x7fd1e013c480 2024-03-25T18:19:54.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.863+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1861173898 0 0) 0x7fd1e00ab0e0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.863+0000 7fd1d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.863+0000 7fd1d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.863+0000 7fd1d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.863+0000 7fd1d7fff640 10 monclient(hunting): my global_id is 4461 2024-03-25T18:19:54.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.863+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e00ab3c0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.867+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2462031954 0 0) 0x7fd1e00a9f30 con 0x7fd1e00b4670 2024-03-25T18:19:54.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.867+0000 7fd1d7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.867+0000 7fd1d7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.867+0000 7fd1d7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.867+0000 7fd1d7fff640 10 monclient(hunting): my global_id is 4481 2024-03-25T18:19:54.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.867+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e00ab0e0 con 0x7fd1e00b4670 2024-03-25T18:19:54.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2863242517 0 0) 0x7fd1e00ab3c0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 >> v1:172.21.15.7:6790/0 conn(0x7fd1e00b4670 legacy=0x7fd1e01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 >> v1:172.21.15.183:6789/0 conn(0x7fd1e013c480 legacy=0x7fd1e014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 1 monclient: found mon.noname-a 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e00a9b70 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1dc001670 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1e7888640 5 monclient: authenticate success, global_id 4461 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1e7888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1d7fff640 10 monclient: dump: 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:54.872 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:54.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.870+0000 7fd1e7888640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:19:54.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1dc0017d0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1d7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1d7fff640 1 -- 172.21.15.7:0/666897150 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1dc001be0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1e7888640 10 monclient: get_monmap_and_config success 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1e7888640 4 set_mon_vals no callback set 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1d7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1d7fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.897+0000 7fd1d7fff640 10 monclient: dump: 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:54.899 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:54.900 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:54.900 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.898+0000 7fd1e7888640 10 monclient: shutdown 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.898+0000 7fd1e7888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.898+0000 7fd1e7888640 1 -- 172.21.15.7:0/666897150 >> v1:172.21.15.7:6789/0 conn(0x7fd1e00b4ff0 legacy=0x7fd1e013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.898+0000 7fd1e7888640 1 -- 172.21.15.7:0/666897150 shutdown_connections 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.898+0000 7fd1e7888640 1 -- 172.21.15.7:0/666897150 >> 172.21.15.7:0/666897150 conn(0x7fd1e00617a0 msgr2=0x7fd1e0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.898+0000 7fd1e7888640 1 -- 172.21.15.7:0/666897150 shutdown_connections 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.898+0000 7fd1e7888640 1 -- 172.21.15.7:0/666897150 wait complete. 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.899+0000 7fd1e7888640 10 monclient: build_initial_monmap 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.899+0000 7fd1e7888640 10 monclient: monmap: 2024-03-25T18:19:54.901 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.900+0000 7fd1e7888640 1 librados: starting msgr at 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.900+0000 7fd1e7888640 1 librados: starting objecter 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.900+0000 7fd1e7888640 1 Processor -- start 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.900+0000 7fd1e7888640 1 -- start start 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.900+0000 7fd1e7888640 1 librados: setting wanted keys 2024-03-25T18:19:54.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.900+0000 7fd1e7888640 1 librados: calling monclient init 2024-03-25T18:19:54.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.900+0000 7fd1e7888640 10 monclient: init 2024-03-25T18:19:54.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 10 monclient: _reopen_session rank -1 2024-03-25T18:19:54.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e00b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e0130cc0 con 0x7fd1e00b4670 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e01445e0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e01448c0 con 0x7fd1e013c480 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 10 monclient(hunting): _renew_subs 2024-03-25T18:19:54.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.901+0000 7fd1e7888640 10 monclient(hunting): authenticate will time out at 7637.005928s 2024-03-25T18:19:54.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.915+0000 7fd1e5e25640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fd1e013c480 0x7fd1e0145780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:34094/0 (socket says 172.21.15.7:34094) 2024-03-25T18:19:54.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.915+0000 7fd1e5e25640 1 -- 172.21.15.7:0/1256365688 learned_addr learned my addr 172.21.15.7:0/1256365688 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:19:54.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.916+0000 7fd1d5ffb640 10 client.?.objecter ms_handle_connect 0x7fd1e013c480 2024-03-25T18:19:54.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.916+0000 7fd1d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.923+0000 7fd1d5ffb640 10 client.?.objecter ms_handle_connect 0x7fd1e00b4ff0 2024-03-25T18:19:54.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.923+0000 7fd1d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.931+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2268729417 0 0) 0x7fd1e01448c0 con 0x7fd1e013c480 2024-03-25T18:19:54.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.932+0000 7fd1d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.932+0000 7fd1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.932+0000 7fd1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.932+0000 7fd1d5ffb640 10 monclient(hunting): my global_id is 4484 2024-03-25T18:19:54.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.932+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c4004eb0 con 0x7fd1e013c480 2024-03-25T18:19:54.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.935+0000 7fd1d5ffb640 10 client.?.objecter ms_handle_connect 0x7fd1e00b4670 2024-03-25T18:19:54.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.935+0000 7fd1d5ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.937+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2463995753 0 0) 0x7fd1e01445e0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.937+0000 7fd1d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.937+0000 7fd1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.937+0000 7fd1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.937+0000 7fd1d5ffb640 10 monclient(hunting): my global_id is 4468 2024-03-25T18:19:54.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.937+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e01448c0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.946+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1112378329 0 0) 0x7fd1e0130cc0 con 0x7fd1e00b4670 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.946+0000 7fd1d5ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:19:54.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.946+0000 7fd1d5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.946+0000 7fd1d5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.946+0000 7fd1d5ffb640 10 monclient(hunting): my global_id is 4464 2024-03-25T18:19:54.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.946+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e01445e0 con 0x7fd1e00b4670 2024-03-25T18:19:54.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.955+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400968761 0 0) 0x7fd1c4004eb0 con 0x7fd1e013c480 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.955+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e0130cc0 con 0x7fd1e013c480 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.955+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1d8001c30 con 0x7fd1e013c480 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.955+0000 7fd1d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.955+0000 7fd1d5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:19:54.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.955+0000 7fd1d5ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.958 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:54.959 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:54.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.955+0000 7fd1d5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2254205216 0 0) 0x7fd1e01448c0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1c4004eb0 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1d0001870 con 0x7fd1e00b4ff0 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 10 monclient(hunting): dump: 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:54.980 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1991398524 0 0) 0x7fd1e0130cc0 con 0x7fd1e013c480 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 >> v1:172.21.15.183:6789/0 conn(0x7fd1e00b4ff0 legacy=0x7fd1e0145070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.978+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 >> v1:172.21.15.7:6789/0 conn(0x7fd1e00b4670 legacy=0x7fd1e013cdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 1 monclient: found mon.c 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e0146530 con 0x7fd1e013c480 2024-03-25T18:19:54.981 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 10 monclient: _finish_auth 0 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1d5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 5 monclient: authenticate success, global_id 4484 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 10 monclient: _renew_subs 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e013ee20 con 0x7fd1e013c480 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 10 monclient: _renew_subs 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e013d6e0 con 0x7fd1e013c480 2024-03-25T18:19:54.982 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.979+0000 7fd1e7888640 1 librados: init done 2024-03-25T18:19:54.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.980+0000 7fd1e7888640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.980+0000 7fd1e7888640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:54.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.980+0000 7fd1e7888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:54.983 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:54.980+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e006dbd0 con 0x7fd1e013c480 2024-03-25T18:19:55.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.205+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1d80020e0 con 0x7fd1e013c480 2024-03-25T18:19:55.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.205+0000 7fd1d5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:19:55.206 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.205+0000 7fd1d77fe640 4 set_mon_vals no callback set 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.303+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1d8003b70 con 0x7fd1e013c480 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.303+0000 7fd1d5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.303+0000 7fd1d5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.303+0000 7fd1d5ffb640 10 monclient: dump: 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:19:55.304 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:19:55.305 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:19:55.305 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:19:55.305 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.304+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fd1d8003d10 con 0x7fd1e013c480 2024-03-25T18:19:55.305 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.304+0000 7fd1e5624640 10 monclient: get_auth_request con 0x7fd1c406df20 auth_method 0 2024-03-25T18:19:55.306 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.305+0000 7fd1d5ffb640 10 client.4484.objecter ms_handle_connect 0x7fd1c406df20 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fd1d8054b40 con 0x7fd1e013c480 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 10 client.4484.objecter ms_dispatch 0x7fd1e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1d8050480 con 0x7fd1e013c480 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:19:55.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.313+0000 7fd1d5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:19:55.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.537+0000 7fd1e7888640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.537+0000 7fd1e7888640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.538+0000 7fd1e7888640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:19:55.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.538+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1e01466c0 con 0x7fd1e013c480 2024-03-25T18:19:55.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.631+0000 7fd1d5ffb640 1 -- 172.21.15.7:0/1256365688 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fd1d8052580 con 0x7fd1e013c480 2024-03-25T18:19:55.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.631+0000 7fd1d5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:19:55.632 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.631+0000 7fd1d5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:19:55.632 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:19:55.633 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:19:55.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 10 librados: watch_flush enter 2024-03-25T18:19:55.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 10 librados: watch_flush exit 2024-03-25T18:19:55.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-25T18:19:55.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 10 client.4484.objecter successfully canceled tick 2024-03-25T18:19:55.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 >> v1:172.21.15.7:6800/1717564767 conn(0x7fd1c406df20 legacy=0x7fd1c40703e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 10 monclient: shutdown 2024-03-25T18:19:55.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:19:55.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 >> v1:172.21.15.7:6790/0 conn(0x7fd1e013c480 legacy=0x7fd1e0145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:19:55.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 shutdown_connections 2024-03-25T18:19:55.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 >> 172.21.15.7:0/1256365688 conn(0x7fd1e00617a0 msgr2=0x7fd1e01405c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:19:55.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 shutdown_connections 2024-03-25T18:19:55.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 1 -- 172.21.15.7:0/1256365688 wait complete. 2024-03-25T18:19:55.636 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:19:55.634+0000 7fd1e7888640 1 librados: shutdown 2024-03-25T18:19:55.648 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:02.454 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.553+0000 7f83b5d9d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.553+0000 7f83b5d9d640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.553+0000 7f83b5d9d640 10 monclient: monmap: 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:02.554174+0000 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:02.554174+0000 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:02.554 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:02.555 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:02.555 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 1 Processor -- start 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 1 -- start start 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient: init 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f83b00b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f83b00b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f83b013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00ab0e0 con 0x7f83b013c480 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00ab3c0 con 0x7f83b00b4ff0 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b00a9f30 con 0x7f83b00b4670 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.554+0000 7f83b5d9d640 10 monclient(hunting): authenticate will time out at 7644.659383s 2024-03-25T18:20:02.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.563+0000 7f83aeffd640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f83b00b4670 0x7f83b01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:49414/0 (socket says 172.21.15.7:49414) 2024-03-25T18:20:02.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.563+0000 7f83aeffd640 1 -- 172.21.15.7:0/2760243525 learned_addr learned my addr 172.21.15.7:0/2760243525 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:02.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.583+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3354295799 0 0) 0x7f83b00a9f30 con 0x7f83b00b4670 2024-03-25T18:20:02.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.583+0000 7f83ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.583+0000 7f83ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.583+0000 7f83ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.583+0000 7f83ae7fc640 10 monclient(hunting): my global_id is 4487 2024-03-25T18:20:02.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.583+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8398003520 con 0x7f83b00b4670 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.588+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1294688979 0 0) 0x7f83b00ab0e0 con 0x7f83b013c480 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.588+0000 7f83ae7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.588+0000 7f83ae7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.588+0000 7f83ae7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.588+0000 7f83ae7fc640 10 monclient(hunting): my global_id is 4467 2024-03-25T18:20:02.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.588+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b00a9f30 con 0x7f83b013c480 2024-03-25T18:20:02.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3835146508 0 0) 0x7f8398003520 con 0x7f83b00b4670 2024-03-25T18:20:02.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 >> v1:172.21.15.183:6789/0 conn(0x7f83b00b4ff0 legacy=0x7f83b013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:20:02.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 >> v1:172.21.15.7:6789/0 conn(0x7f83b013c480 legacy=0x7f83b014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 1 monclient: found mon.noname-b 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b00a9b70 con 0x7f83b00b4670 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83ae7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83b5d9d640 5 monclient: authenticate success, global_id 4487 2024-03-25T18:20:02.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.590+0000 7f83b5d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:02.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.593+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f83a4001650 con 0x7f83b00b4670 2024-03-25T18:20:02.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.593+0000 7f83ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.593+0000 7f83ae7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:02.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.593+0000 7f83ae7fc640 10 monclient: dump: 2024-03-25T18:20:02.594 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:02.594 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:02.594 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:02.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.593+0000 7f83b5d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:02.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83a40018a0 con 0x7f83b00b4670 2024-03-25T18:20:02.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83ae7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83ae7fc640 1 -- 172.21.15.7:0/2760243525 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f83a4001c30 con 0x7f83b00b4670 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 4 set_mon_vals no callback set 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83ae7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83ae7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83ae7fc640 10 monclient: dump: 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:02.626 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:02.627 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:02.627 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:02.627 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 10 monclient: shutdown 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 1 -- 172.21.15.7:0/2760243525 >> v1:172.21.15.7:6790/0 conn(0x7f83b00b4670 legacy=0x7f83b01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 1 -- 172.21.15.7:0/2760243525 shutdown_connections 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 1 -- 172.21.15.7:0/2760243525 >> 172.21.15.7:0/2760243525 conn(0x7f83b00617a0 msgr2=0x7f83b0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 1 -- 172.21.15.7:0/2760243525 shutdown_connections 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.625+0000 7f83b5d9d640 1 -- 172.21.15.7:0/2760243525 wait complete. 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.626+0000 7f83b5d9d640 10 monclient: build_initial_monmap 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.626+0000 7f83b5d9d640 10 monclient: monmap: 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:02.629 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.626+0000 7f83b5d9d640 1 librados: starting msgr at 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.626+0000 7f83b5d9d640 1 librados: starting objecter 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 1 Processor -- start 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 1 -- start start 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 1 librados: setting wanted keys 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 1 librados: calling monclient init 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient: init 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:02.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f83b00b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f83b00b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f83b013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b0143fe0 con 0x7f83b00b4ff0 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b01447c0 con 0x7f83b00b4670 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83b0144a30 con 0x7f83b013c480 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:02.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.627+0000 7f83b5d9d640 10 monclient(hunting): authenticate will time out at 7644.732595s 2024-03-25T18:20:02.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.639+0000 7f83af7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f83b00b4670 0x7f83b01438d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:55462/0 (socket says 172.21.15.7:55462) 2024-03-25T18:20:02.640 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.639+0000 7f83af7fe640 1 -- 172.21.15.7:0/3244305178 learned_addr learned my addr 172.21.15.7:0/3244305178 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:02.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.643+0000 7f8397fff640 10 client.?.objecter ms_handle_connect 0x7f83b00b4670 2024-03-25T18:20:02.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.643+0000 7f8397fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.644+0000 7f8397fff640 10 client.?.objecter ms_handle_connect 0x7f83b00b4ff0 2024-03-25T18:20:02.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.644+0000 7f8397fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.650+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 380498307 0 0) 0x7f83b01447c0 con 0x7f83b00b4670 2024-03-25T18:20:02.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.650+0000 7f8397fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.650+0000 7f8397fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.650+0000 7f8397fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.650+0000 7f8397fff640 10 monclient(hunting): my global_id is 4471 2024-03-25T18:20:02.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.650+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f839c004eb0 con 0x7f83b00b4670 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3483905675 0 0) 0x7f83b0143fe0 con 0x7f83b00b4ff0 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 10 monclient(hunting): my global_id is 4470 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b01447c0 con 0x7f83b00b4ff0 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 10 client.?.objecter ms_handle_connect 0x7f83b013c480 2024-03-25T18:20:02.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.654+0000 7f8397fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:02.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.663+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2721848707 0 0) 0x7f83b0144a30 con 0x7f83b013c480 2024-03-25T18:20:02.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.663+0000 7f8397fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:02.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.663+0000 7f8397fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:02.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.663+0000 7f8397fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:02.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.663+0000 7f8397fff640 10 monclient(hunting): my global_id is 4490 2024-03-25T18:20:02.664 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.663+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b0143fe0 con 0x7f83b013c480 2024-03-25T18:20:02.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.670+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2809350818 0 0) 0x7f839c004eb0 con 0x7f83b00b4670 2024-03-25T18:20:02.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.670+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83b0144a30 con 0x7f83b00b4670 2024-03-25T18:20:02.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.670+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f83a0001960 con 0x7f83b00b4670 2024-03-25T18:20:02.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.670+0000 7f8397fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.670+0000 7f8397fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:02.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.670+0000 7f8397fff640 10 monclient(hunting): dump: 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:02.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.670+0000 7f8397fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.673+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1262374931 0 0) 0x7f83b01447c0 con 0x7f83b00b4ff0 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.673+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f839c004eb0 con 0x7f83b00b4ff0 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.673+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f83a4001bb0 con 0x7f83b00b4ff0 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.673+0000 7f8397fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.673+0000 7f8397fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.673+0000 7f8397fff640 10 monclient(hunting): dump: 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.674 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.675 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.675 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:02.675 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:02.675 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:02.675 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:02.675 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4123114455 0 0) 0x7f83b0144a30 con 0x7f83b00b4670 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 >> v1:172.21.15.7:6790/0 conn(0x7f83b013c480 legacy=0x7f83b0145cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 >> v1:172.21.15.7:6789/0 conn(0x7f83b00b4ff0 legacy=0x7f83b01455a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 1 monclient: found mon.b 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83b0144d10 con 0x7f83b00b4670 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f8397fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:02.684 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 5 monclient: authenticate success, global_id 4471 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 10 monclient: _renew_subs 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83b013c0d0 con 0x7f83b00b4670 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 10 client.4471.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 10 monclient: _renew_subs 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83b01e3bc0 con 0x7f83b00b4670 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.682+0000 7f83b5d9d640 1 librados: init done 2024-03-25T18:20:02.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.684+0000 7f83b5d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.684+0000 7f83b5d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.684+0000 7f83b5d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:02.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.684+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b006dbd0 con 0x7f83b00b4670 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.954+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83a00035f0 con 0x7f83b00b4670 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.954+0000 7f8397fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.954+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f83a0003940 con 0x7f83b00b4670 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.954+0000 7f8397fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.954+0000 7f8397fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.954+0000 7f8397fff640 10 monclient: dump: 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.955 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.954+0000 7f83adffb640 4 set_mon_vals no callback set 2024-03-25T18:20:02.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.955+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f83a0003b20 con 0x7f83b00b4670 2024-03-25T18:20:02.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.955+0000 7f83aeffd640 10 monclient: get_auth_request con 0x7f839c06dee0 auth_method 0 2024-03-25T18:20:02.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.956+0000 7f8397fff640 10 client.4471.objecter ms_handle_connect 0x7f839c06dee0 2024-03-25T18:20:02.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.959+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f83a00547d0 con 0x7f83b00b4670 2024-03-25T18:20:02.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.959+0000 7f8397fff640 10 client.4471.objecter ms_dispatch 0x7f83b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:02.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.959+0000 7f8397fff640 3 client.4471.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:02.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.959+0000 7f8397fff640 3 client.4471.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:02.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.959+0000 7f8397fff640 20 client.4471.objecter dump_active .. 0 homeless 2024-03-25T18:20:02.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.960+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83a00502e0 con 0x7f83b00b4670 2024-03-25T18:20:02.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.960+0000 7f8397fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:02.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:02.960+0000 7f8397fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:03.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.209+0000 7f83b5d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:03.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.209+0000 7f83b5d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:03.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.209+0000 7f83b5d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:03.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.209+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83b0146790 con 0x7f83b00b4670 2024-03-25T18:20:03.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.227+0000 7f8397fff640 1 -- 172.21.15.7:0/3244305178 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f83a0052210 con 0x7f83b00b4670 2024-03-25T18:20:03.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.227+0000 7f8397fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:03.228 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.227+0000 7f8397fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:03.229 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:03.229 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:03.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 10 librados: watch_flush enter 2024-03-25T18:20:03.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 10 librados: watch_flush exit 2024-03-25T18:20:03.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 20 client.4471.objecter shutdown clearing up homeless session... 2024-03-25T18:20:03.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 10 client.4471.objecter successfully canceled tick 2024-03-25T18:20:03.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 >> v1:172.21.15.7:6800/1717564767 conn(0x7f839c06dee0 legacy=0x7f839c0703a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 10 monclient: shutdown 2024-03-25T18:20:03.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:03.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 >> v1:172.21.15.183:6789/0 conn(0x7f83b00b4670 legacy=0x7f83b01438d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:03.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 shutdown_connections 2024-03-25T18:20:03.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 >> 172.21.15.7:0/3244305178 conn(0x7f83b00617a0 msgr2=0x7f83b00b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:03.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 shutdown_connections 2024-03-25T18:20:03.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 1 -- 172.21.15.7:0/3244305178 wait complete. 2024-03-25T18:20:03.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:03.230+0000 7f83b5d9d640 1 librados: shutdown 2024-03-25T18:20:03.244 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:10.051 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:10.168 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.167+0000 7f27dae7b640 10 monclient: get_monmap_and_config 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.167+0000 7f27dae7b640 10 monclient: build_initial_monmap 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.167+0000 7f27dae7b640 10 monclient: monmap: 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:10.168940+0000 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:10.168940+0000 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:10.169 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.168+0000 7f27dae7b640 1 Processor -- start 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.168+0000 7f27dae7b640 1 -- start start 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.168+0000 7f27dae7b640 10 monclient: init 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 10 monclient(hunting): picked mon.noname-c con 0x7f27d40b49f0 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 10 monclient(hunting): picked mon.noname-b con 0x7f27d40b4dd0 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 10 monclient(hunting): picked mon.noname-a con 0x7f27d40a9a90 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:10.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d40aa070 con 0x7f27d40a9a90 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d40aaf40 con 0x7f27d40b49f0 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d40ab220 con 0x7f27d40b4dd0 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:10.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.169+0000 7f27dae7b640 10 monclient(hunting): authenticate will time out at 7652.274200s 2024-03-25T18:20:10.186 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.185+0000 7f27d3fff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f27d40b4dd0 0x7f27d4142520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:53892/0 (socket says 172.21.15.7:53892) 2024-03-25T18:20:10.186 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.185+0000 7f27d3fff640 1 -- 172.21.15.7:0/1832002899 learned_addr learned my addr 172.21.15.7:0/1832002899 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:10.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.198+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41610955 0 0) 0x7f27d40ab220 con 0x7f27d40b4dd0 2024-03-25T18:20:10.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.198+0000 7f27d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.198+0000 7f27d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.198+0000 7f27d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.198+0000 7f27d37fe640 10 monclient(hunting): my global_id is 4493 2024-03-25T18:20:10.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.198+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27c4003520 con 0x7f27d40b4dd0 2024-03-25T18:20:10.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.203+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4231232663 0 0) 0x7f27d40aaf40 con 0x7f27d40b49f0 2024-03-25T18:20:10.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.203+0000 7f27d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.203+0000 7f27d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.203+0000 7f27d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.203+0000 7f27d37fe640 10 monclient(hunting): my global_id is 4474 2024-03-25T18:20:10.205 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.203+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d40ab220 con 0x7f27d40b49f0 2024-03-25T18:20:10.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.209+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1082336151 0 0) 0x7f27d40aa070 con 0x7f27d40a9a90 2024-03-25T18:20:10.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.209+0000 7f27d37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.209+0000 7f27d37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.209+0000 7f27d37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.209+0000 7f27d37fe640 10 monclient(hunting): my global_id is 4473 2024-03-25T18:20:10.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.209+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d40aaf40 con 0x7f27d40a9a90 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2951674634 0 0) 0x7f27d40ab220 con 0x7f27d40b49f0 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 >> v1:172.21.15.7:6790/0 conn(0x7f27d40b4dd0 legacy=0x7f27d4142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 >> v1:172.21.15.7:6789/0 conn(0x7f27d40a9a90 legacy=0x7f27d4142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 1 monclient: found mon.noname-c 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:20:10.222 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d413c410 con 0x7f27d40b49f0 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27c8001670 con 0x7f27d40b49f0 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.220+0000 7f27d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.221+0000 7f27d37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.221+0000 7f27d37fe640 10 monclient: dump: 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:10.223 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:10.224 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:10.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.221+0000 7f27dae7b640 5 monclient: authenticate success, global_id 4474 2024-03-25T18:20:10.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.221+0000 7f27dae7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27c80017d0 con 0x7f27d40b49f0 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27d37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27d37fe640 1 -- 172.21.15.7:0/1832002899 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27c8001be0 con 0x7f27d40b49f0 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27dae7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27dae7b640 4 set_mon_vals no callback set 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27d37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27d37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.251+0000 7f27d37fe640 10 monclient: dump: 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:10.252 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:10.253 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:10.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.252+0000 7f27dae7b640 10 monclient: shutdown 2024-03-25T18:20:10.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.252+0000 7f27dae7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.255 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.252+0000 7f27dae7b640 1 -- 172.21.15.7:0/1832002899 >> v1:172.21.15.183:6789/0 conn(0x7f27d40b49f0 legacy=0x7f27d413bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.252+0000 7f27dae7b640 1 -- 172.21.15.7:0/1832002899 shutdown_connections 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.252+0000 7f27dae7b640 1 -- 172.21.15.7:0/1832002899 >> 172.21.15.7:0/1832002899 conn(0x7f27d40615c0 msgr2=0x7f27d40619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.252+0000 7f27dae7b640 1 -- 172.21.15.7:0/1832002899 shutdown_connections 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.252+0000 7f27dae7b640 1 -- 172.21.15.7:0/1832002899 wait complete. 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 10 monclient: build_initial_monmap 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 10 monclient: monmap: 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:10.256 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 1 librados: starting msgr at 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 1 librados: starting objecter 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 1 Processor -- start 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 1 -- start start 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 1 librados: setting wanted keys 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 1 librados: calling monclient init 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.253+0000 7f27dae7b640 10 monclient: init 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 10 monclient(hunting): picked mon.noname-a con 0x7f27d40a9a90 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 10 monclient(hunting): picked mon.noname-c con 0x7f27d40b49f0 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 10 monclient(hunting): picked mon.noname-b con 0x7f27d40b4dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d413da70 con 0x7f27d40a9a90 2024-03-25T18:20:10.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d413dce0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f27d413bf20 con 0x7f27d40b49f0 2024-03-25T18:20:10.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:10.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.254+0000 7f27dae7b640 10 monclient(hunting): authenticate will time out at 7652.359199s 2024-03-25T18:20:10.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.258+0000 7f27d9418640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f27d40b4dd0 0x7f27d41e1ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:60144/0 (socket says 172.21.15.7:60144) 2024-03-25T18:20:10.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.258+0000 7f27d9418640 1 -- 172.21.15.7:0/2587882504 learned_addr learned my addr 172.21.15.7:0/2587882504 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:10.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.262+0000 7f27d17fa640 10 client.?.objecter ms_handle_connect 0x7f27d40b4dd0 2024-03-25T18:20:10.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.262+0000 7f27d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.271+0000 7f27d17fa640 10 client.?.objecter ms_handle_connect 0x7f27d40b49f0 2024-03-25T18:20:10.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.271+0000 7f27d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.277+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4230946936 0 0) 0x7f27d413dce0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.277+0000 7f27d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.277+0000 7f27d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.277+0000 7f27d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.277+0000 7f27d17fa640 10 monclient(hunting): my global_id is 4477 2024-03-25T18:20:10.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.277+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b8004eb0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.285+0000 7f27d17fa640 10 client.?.objecter ms_handle_connect 0x7f27d40a9a90 2024-03-25T18:20:10.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.285+0000 7f27d17fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.297+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 649519120 0 0) 0x7f27b8004eb0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.297+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f27d413dce0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.297+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27cc001d10 con 0x7f27d40b4dd0 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.297+0000 7f27d17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.297+0000 7f27d17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.297+0000 7f27d17fa640 10 monclient(hunting): dump: 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.298 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.299 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:10.299 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:10.299 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:10.299 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:10.299 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:10.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.297+0000 7f27d17fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:10.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.307+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1208894976 0 0) 0x7f27d413da70 con 0x7f27d40a9a90 2024-03-25T18:20:10.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.307+0000 7f27d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.307+0000 7f27d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.307+0000 7f27d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.307+0000 7f27d17fa640 10 monclient(hunting): my global_id is 4476 2024-03-25T18:20:10.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.307+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27b8004eb0 con 0x7f27d40a9a90 2024-03-25T18:20:10.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.310+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2192366336 0 0) 0x7f27d413bf20 con 0x7f27d40b49f0 2024-03-25T18:20:10.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.310+0000 7f27d17fa640 10 monclient(hunting): _negotiate 2024-03-25T18:20:10.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.310+0000 7f27d17fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:10.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.310+0000 7f27d17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:10.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.310+0000 7f27d17fa640 10 monclient(hunting): my global_id is 4496 2024-03-25T18:20:10.311 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.310+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f27d413da70 con 0x7f27d40b49f0 2024-03-25T18:20:10.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3043372885 0 0) 0x7f27d413dce0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 >> v1:172.21.15.7:6790/0 conn(0x7f27d40b49f0 legacy=0x7f27d4143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 >> v1:172.21.15.7:6789/0 conn(0x7f27d40a9a90 legacy=0x7f27d41436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 10 monclient: _finish_hunting 0 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 1 monclient: found mon.b 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d413f460 con 0x7f27d40b4dd0 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 10 monclient: _finish_auth 0 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 10 monclient: _check_auth_tickets 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27d17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 5 monclient: authenticate success, global_id 4477 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 10 monclient: _renew_subs 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d4146350 con 0x7f27d40b4dd0 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:10.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 10 monclient: _renew_subs 2024-03-25T18:20:10.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:10.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d413cb60 con 0x7f27d40b4dd0 2024-03-25T18:20:10.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.324+0000 7f27dae7b640 1 librados: init done 2024-03-25T18:20:10.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.325+0000 7f27dae7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.325+0000 7f27dae7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.325+0000 7f27dae7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:10.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.325+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d413c0b0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.609 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.608+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f27cc003970 con 0x7f27d40b4dd0 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.608+0000 7f27d17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.608+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f27cc003dc0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.608+0000 7f27d17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.608+0000 7f27d17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.608+0000 7f27d17fa640 10 monclient: dump: 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:10.610 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.608+0000 7f27d2ffd640 4 set_mon_vals no callback set 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.609+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f27cc003fa0 con 0x7f27d40b4dd0 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.610+0000 7f27d8c17640 10 monclient: get_auth_request con 0x7f27b806de80 auth_method 0 2024-03-25T18:20:10.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.610+0000 7f27d17fa640 10 client.4477.objecter ms_handle_connect 0x7f27b806de80 2024-03-25T18:20:10.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f27cc054c20 con 0x7f27d40b4dd0 2024-03-25T18:20:10.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 10 client.4477.objecter ms_dispatch 0x7f27d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:10.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:10.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:10.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-25T18:20:10.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f27cc050730 con 0x7f27d40b4dd0 2024-03-25T18:20:10.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:10.615 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.613+0000 7f27d17fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:10.841 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.840+0000 7f27dae7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.841 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.840+0000 7f27dae7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.841 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.840+0000 7f27dae7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:10.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.840+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f27d404f730 con 0x7f27d40b4dd0 2024-03-25T18:20:10.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.873+0000 7f27d17fa640 1 -- 172.21.15.7:0/2587882504 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f27cc052660 con 0x7f27d40b4dd0 2024-03-25T18:20:10.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.873+0000 7f27d17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:10.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.873+0000 7f27d17fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:10.875 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:10.875 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 10 librados: watch_flush enter 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 10 librados: watch_flush exit 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 10 client.4477.objecter successfully canceled tick 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 >> v1:172.21.15.7:6800/1717564767 conn(0x7f27b806de80 legacy=0x7f27b8070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 10 monclient: shutdown 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 >> v1:172.21.15.183:6789/0 conn(0x7f27d40b4dd0 legacy=0x7f27d41e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 shutdown_connections 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 >> 172.21.15.7:0/2587882504 conn(0x7f27d40615c0 msgr2=0x7f27d40b5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 shutdown_connections 2024-03-25T18:20:10.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 1 -- 172.21.15.7:0/2587882504 wait complete. 2024-03-25T18:20:10.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:10.876+0000 7f27dae7b640 1 librados: shutdown 2024-03-25T18:20:10.890 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:14.104 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:20.741 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.151 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:20.788 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.198 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:20.835 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.222 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:20.849 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.225 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:20.852 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.235 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:20.872 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.243 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:20.880 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.253 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:20.879 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:14.281 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:20.918 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:17.697 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:17.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.813+0000 7f8544123640 10 monclient: get_monmap_and_config 2024-03-25T18:20:17.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.813+0000 7f8544123640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.813+0000 7f8544123640 10 monclient: monmap: 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:17.814357+0000 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:17.814357+0000 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:17.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 1 Processor -- start 2024-03-25T18:20:17.816 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 1 -- start start 2024-03-25T18:20:17.816 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient: init 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient(hunting): picked mon.noname-a con 0x7f853c0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient(hunting): picked mon.noname-c con 0x7f853c0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient(hunting): picked mon.noname-b con 0x7f853c13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0ab0e0 con 0x7f853c0b4ff0 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0ab3c0 con 0x7f853c0b4670 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c0a9f30 con 0x7f853c13c480 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.814+0000 7f8544123640 10 monclient(hunting): authenticate will time out at 7659.919494s 2024-03-25T18:20:17.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.819+0000 7f8541ebf640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f853c0b4ff0 0x7f853c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38450/0 (socket says 172.21.15.7:38450) 2024-03-25T18:20:17.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.819+0000 7f8541ebf640 1 -- 172.21.15.7:0/1736687120 learned_addr learned my addr 172.21.15.7:0/1736687120 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:17.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.833+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 668922944 0 0) 0x7f853c0ab3c0 con 0x7f853c0b4670 2024-03-25T18:20:17.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.833+0000 7f8540ebd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.833+0000 7f8540ebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.833+0000 7f8540ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.833+0000 7f8540ebd640 10 monclient(hunting): my global_id is 4480 2024-03-25T18:20:17.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.833+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8528003520 con 0x7f853c0b4670 2024-03-25T18:20:17.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.836+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2138943437 0 0) 0x7f853c0ab0e0 con 0x7f853c0b4ff0 2024-03-25T18:20:17.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.836+0000 7f8540ebd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.836+0000 7f8540ebd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.836+0000 7f8540ebd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.836+0000 7f8540ebd640 10 monclient(hunting): my global_id is 4479 2024-03-25T18:20:17.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.836+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f853c0ab3c0 con 0x7f853c0b4ff0 2024-03-25T18:20:17.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2125900812 0 0) 0x7f8528003520 con 0x7f853c0b4670 2024-03-25T18:20:17.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 >> v1:172.21.15.7:6790/0 conn(0x7f853c13c480 legacy=0x7f853c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 >> v1:172.21.15.7:6789/0 conn(0x7f853c0b4ff0 legacy=0x7f853c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 1 monclient: found mon.noname-c 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c0a9b70 con 0x7f853c0b4670 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8540ebd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8544123640 5 monclient: authenticate success, global_id 4480 2024-03-25T18:20:17.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.837+0000 7f8544123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.843+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f852c001650 con 0x7f853c0b4670 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.843+0000 7f8540ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.843+0000 7f8540ebd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.843+0000 7f8540ebd640 10 monclient: dump: 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.844 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:17.845 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:17.845 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:17.845 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:17.845 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:17.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.843+0000 7f8544123640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f852c0018a0 con 0x7f853c0b4670 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8540ebd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8540ebd640 1 -- 172.21.15.7:0/1736687120 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f852c001c30 con 0x7f853c0b4670 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8540ebd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8540ebd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8540ebd640 10 monclient: dump: 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:17.854 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8544123640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:17.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.853+0000 7f8544123640 4 set_mon_vals no callback set 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.854+0000 7f8544123640 10 monclient: shutdown 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.854+0000 7f8544123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.854+0000 7f8544123640 1 -- 172.21.15.7:0/1736687120 >> v1:172.21.15.183:6789/0 conn(0x7f853c0b4670 legacy=0x7f853c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.854+0000 7f8544123640 1 -- 172.21.15.7:0/1736687120 shutdown_connections 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.854+0000 7f8544123640 1 -- 172.21.15.7:0/1736687120 >> 172.21.15.7:0/1736687120 conn(0x7f853c0617a0 msgr2=0x7f853c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.854+0000 7f8544123640 1 -- 172.21.15.7:0/1736687120 shutdown_connections 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.854+0000 7f8544123640 1 -- 172.21.15.7:0/1736687120 wait complete. 2024-03-25T18:20:17.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 10 monclient: build_initial_monmap 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 10 monclient: monmap: 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:17.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 1 librados: starting msgr at 2024-03-25T18:20:17.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 1 librados: starting objecter 2024-03-25T18:20:17.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 1 Processor -- start 2024-03-25T18:20:17.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 1 -- start start 2024-03-25T18:20:17.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 1 librados: setting wanted keys 2024-03-25T18:20:17.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 1 librados: calling monclient init 2024-03-25T18:20:17.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.855+0000 7f8544123640 10 monclient: init 2024-03-25T18:20:17.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.856+0000 7f8544123640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:17.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.856+0000 7f8544123640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:17.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.856+0000 7f8544123640 10 monclient(hunting): picked mon.noname-a con 0x7f853c0b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:17.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.856+0000 7f8544123640 10 monclient(hunting): picked mon.noname-b con 0x7f853c0b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:17.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.856+0000 7f8544123640 10 monclient(hunting): picked mon.noname-c con 0x7f853c13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:17.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.856+0000 7f8544123640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c144a90 con 0x7f853c0b4670 2024-03-25T18:20:17.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.857+0000 7f8544123640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c143660 con 0x7f853c0b4ff0 2024-03-25T18:20:17.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.857+0000 7f8544123640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f853c13e010 con 0x7f853c13c480 2024-03-25T18:20:17.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.857+0000 7f8544123640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:17.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.857+0000 7f8544123640 10 monclient(hunting): authenticate will time out at 7659.961676s 2024-03-25T18:20:17.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.860+0000 7f8541ebf640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f853c0b4670 0x7f853c145450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:38470/0 (socket says 172.21.15.7:38470) 2024-03-25T18:20:17.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.860+0000 7f8541ebf640 1 -- 172.21.15.7:0/3378839404 learned_addr learned my addr 172.21.15.7:0/3378839404 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:17.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.861+0000 7f85327fc640 10 client.?.objecter ms_handle_connect 0x7f853c0b4670 2024-03-25T18:20:17.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.861+0000 7f85327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.877+0000 7f85327fc640 10 client.?.objecter ms_handle_connect 0x7f853c13c480 2024-03-25T18:20:17.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.877+0000 7f85327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.884+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1034174905 0 0) 0x7f853c144a90 con 0x7f853c0b4670 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.884+0000 7f85327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.884+0000 7f85327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.884+0000 7f85327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.884+0000 7f85327fc640 10 monclient(hunting): my global_id is 4482 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.884+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8520004eb0 con 0x7f853c0b4670 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.885+0000 7f85327fc640 10 client.?.objecter ms_handle_connect 0x7f853c0b4ff0 2024-03-25T18:20:17.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.885+0000 7f85327fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:17.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.890+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3405844055 0 0) 0x7f853c13e010 con 0x7f853c13c480 2024-03-25T18:20:17.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.890+0000 7f85327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.890+0000 7f85327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.890+0000 7f85327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.890+0000 7f85327fc640 10 monclient(hunting): my global_id is 4502 2024-03-25T18:20:17.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.890+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f853c144a90 con 0x7f853c13c480 2024-03-25T18:20:17.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.896+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070155231 0 0) 0x7f853c143660 con 0x7f853c0b4ff0 2024-03-25T18:20:17.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.896+0000 7f85327fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:17.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.896+0000 7f85327fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:17.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.896+0000 7f85327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:17.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.896+0000 7f85327fc640 10 monclient(hunting): my global_id is 4483 2024-03-25T18:20:17.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.896+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f853c13e010 con 0x7f853c0b4ff0 2024-03-25T18:20:17.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.914+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 518836887 0 0) 0x7f853c144a90 con 0x7f853c13c480 2024-03-25T18:20:17.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.914+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f853c143660 con 0x7f853c13c480 2024-03-25T18:20:17.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.914+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8534001b00 con 0x7f853c13c480 2024-03-25T18:20:17.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.914+0000 7f85327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.914+0000 7f85327fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:17.915 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.914+0000 7f85327fc640 10 monclient(hunting): dump: 2024-03-25T18:20:17.915 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:17.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.914+0000 7f85327fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.915+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1708336137 0 0) 0x7f8520004eb0 con 0x7f853c0b4670 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.915+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f853c144a90 con 0x7f853c0b4670 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.915+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f85380019b0 con 0x7f853c0b4670 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.915+0000 7f85327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.915+0000 7f85327fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.915+0000 7f85327fc640 10 monclient(hunting): dump: 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:17.917 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:17.918 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:17.918 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:17.918 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:17.918 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:17.918 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 593828282 0 0) 0x7f853c143660 con 0x7f853c13c480 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 >> v1:172.21.15.183:6789/0 conn(0x7f853c0b4ff0 legacy=0x7f853c13c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 >> v1:172.21.15.7:6789/0 conn(0x7f853c0b4670 legacy=0x7f853c145450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 1 monclient: found mon.c 2024-03-25T18:20:17.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c144cc0 con 0x7f853c13c480 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f85327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 5 monclient: authenticate success, global_id 4502 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 10 monclient: _renew_subs 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f853c143af0 con 0x7f853c13c480 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 10 client.4502.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 10 monclient: _renew_subs 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:17.924 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f853c146500 con 0x7f853c13c480 2024-03-25T18:20:17.925 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.922+0000 7f8544123640 1 librados: init done 2024-03-25T18:20:17.925 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.923+0000 7f8544123640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.925 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.923+0000 7f8544123640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:17.925 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.923+0000 7f8544123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:17.925 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:17.923+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f853c06dbd0 con 0x7f853c13c480 2024-03-25T18:20:18.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.255+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8534001fb0 con 0x7f853c13c480 2024-03-25T18:20:18.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.255+0000 7f85327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:18.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.255+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8534003b10 con 0x7f853c13c480 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.255+0000 7f85327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.255+0000 7f85327fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.255+0000 7f85327fc640 10 monclient: dump: 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:18.257 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:18.258 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:18.258 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:18.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.255+0000 7f8533fff640 4 set_mon_vals no callback set 2024-03-25T18:20:18.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.256+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f8534003d30 con 0x7f853c13c480 2024-03-25T18:20:18.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.257+0000 7f8541ebf640 10 monclient: get_auth_request con 0x7f852006df70 auth_method 0 2024-03-25T18:20:18.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.257+0000 7f85327fc640 10 client.4502.objecter ms_handle_connect 0x7f852006df70 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.265+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f85340536a0 con 0x7f853c13c480 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.265+0000 7f85327fc640 10 client.4502.objecter ms_dispatch 0x7f853c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.265+0000 7f85327fc640 3 client.4502.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.265+0000 7f85327fc640 3 client.4502.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.265+0000 7f85327fc640 20 client.4502.objecter dump_active .. 0 homeless 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.266+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f853404f1b0 con 0x7f853c13c480 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.266+0000 7f85327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:18.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.266+0000 7f85327fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:18.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.512+0000 7f8544123640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.512+0000 7f8544123640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.512+0000 7f8544123640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:18.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.512+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f853c0688c0 con 0x7f853c13c480 2024-03-25T18:20:18.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.538+0000 7f85327fc640 1 -- 172.21.15.7:0/3378839404 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f85340510e0 con 0x7f853c13c480 2024-03-25T18:20:18.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.538+0000 7f85327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:18.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.538+0000 7f85327fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:18.540 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:18.540 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:18.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.540+0000 7f8544123640 10 librados: watch_flush enter 2024-03-25T18:20:18.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.540+0000 7f8544123640 10 librados: watch_flush exit 2024-03-25T18:20:18.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.540+0000 7f8544123640 20 client.4502.objecter shutdown clearing up homeless session... 2024-03-25T18:20:18.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.540+0000 7f8544123640 10 client.4502.objecter successfully canceled tick 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 >> v1:172.21.15.7:6800/1717564767 conn(0x7f852006df70 legacy=0x7f8520070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 10 monclient: shutdown 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 >> v1:172.21.15.7:6790/0 conn(0x7f853c13c480 legacy=0x7f853c1e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 shutdown_connections 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 >> 172.21.15.7:0/3378839404 conn(0x7f853c0617a0 msgr2=0x7f853c1407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 shutdown_connections 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 1 -- 172.21.15.7:0/3378839404 wait complete. 2024-03-25T18:20:18.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:18.541+0000 7f8544123640 1 librados: shutdown 2024-03-25T18:20:18.554 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:25.361 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.479+0000 7fbf05dd0640 10 monclient: get_monmap_and_config 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.479+0000 7fbf05dd0640 10 monclient: build_initial_monmap 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.479+0000 7fbf05dd0640 10 monclient: monmap: 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:25.480 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:25.480636+0000 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:25.480636+0000 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:25.481 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.480+0000 7fbf05dd0640 1 Processor -- start 2024-03-25T18:20:25.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.480+0000 7fbf05dd0640 1 -- start start 2024-03-25T18:20:25.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.480+0000 7fbf05dd0640 10 monclient: init 2024-03-25T18:20:25.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:25.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:25.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 10 monclient(hunting): picked mon.noname-a con 0x7fbf000b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:25.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 10 monclient(hunting): picked mon.noname-b con 0x7fbf000b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:25.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 10 monclient(hunting): picked mon.noname-c con 0x7fbf0013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:25.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000ab0e0 con 0x7fbf000b4ff0 2024-03-25T18:20:25.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000ab3c0 con 0x7fbf0013c480 2024-03-25T18:20:25.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000a9f30 con 0x7fbf000b4670 2024-03-25T18:20:25.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:25.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.481+0000 7fbf05dd0640 10 monclient(hunting): authenticate will time out at 7667.585999s 2024-03-25T18:20:25.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.484+0000 7fbefeffd640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fbf000b4670 0x7fbf001425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:34428/0 (socket says 172.21.15.7:34428) 2024-03-25T18:20:25.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.484+0000 7fbefeffd640 1 -- 172.21.15.7:0/936617967 learned_addr learned my addr 172.21.15.7:0/936617967 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:25.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.498+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322509825 0 0) 0x7fbf000a9f30 con 0x7fbf000b4670 2024-03-25T18:20:25.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.498+0000 7fbefe7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.498+0000 7fbefe7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.498+0000 7fbefe7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.498+0000 7fbefe7fc640 10 monclient(hunting): my global_id is 4505 2024-03-25T18:20:25.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.498+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbee8003520 con 0x7fbf000b4670 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 206652303 0 0) 0x7fbee8003520 con 0x7fbf000b4670 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 >> v1:172.21.15.183:6789/0 conn(0x7fbf0013c480 legacy=0x7fbf0014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 >> v1:172.21.15.7:6789/0 conn(0x7fbf000b4ff0 legacy=0x7fbf0013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 1 monclient: found mon.noname-b 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000a9b70 con 0x7fbf000b4670 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbef4001690 con 0x7fbf000b4670 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbf05dd0640 5 monclient: authenticate success, global_id 4505 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbf05dd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:25.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbefe7fc640 10 monclient: dump: 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.504+0000 7fbf05dd0640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:25.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef40017f0 con 0x7fbf000b4670 2024-03-25T18:20:25.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbefe7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:25.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbefe7fc640 1 -- 172.21.15.7:0/936617967 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbef4001c00 con 0x7fbf000b4670 2024-03-25T18:20:25.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbefe7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbefe7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbefe7fc640 10 monclient: dump: 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbf05dd0640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:25.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.513+0000 7fbf05dd0640 4 set_mon_vals no callback set 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.514+0000 7fbf05dd0640 10 monclient: shutdown 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.514+0000 7fbf05dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.514+0000 7fbf05dd0640 1 -- 172.21.15.7:0/936617967 >> v1:172.21.15.7:6790/0 conn(0x7fbf000b4670 legacy=0x7fbf001425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.514+0000 7fbf05dd0640 1 -- 172.21.15.7:0/936617967 shutdown_connections 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.514+0000 7fbf05dd0640 1 -- 172.21.15.7:0/936617967 >> 172.21.15.7:0/936617967 conn(0x7fbf000617a0 msgr2=0x7fbf00061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.514+0000 7fbf05dd0640 1 -- 172.21.15.7:0/936617967 shutdown_connections 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.514+0000 7fbf05dd0640 1 -- 172.21.15.7:0/936617967 wait complete. 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.515+0000 7fbf05dd0640 10 monclient: build_initial_monmap 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.515+0000 7fbf05dd0640 10 monclient: monmap: 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:25.517 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.515+0000 7fbf05dd0640 1 librados: starting msgr at 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.515+0000 7fbf05dd0640 1 librados: starting objecter 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.515+0000 7fbf05dd0640 1 Processor -- start 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 1 -- start start 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 1 librados: setting wanted keys 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 1 librados: calling monclient init 2024-03-25T18:20:25.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 10 monclient: init 2024-03-25T18:20:25.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:25.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 10 monclient(hunting): picked mon.noname-b con 0x7fbf000b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.516+0000 7fbf05dd0640 10 monclient(hunting): picked mon.noname-a con 0x7fbf000b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.517+0000 7fbf05dd0640 10 monclient(hunting): picked mon.noname-c con 0x7fbf0013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.517+0000 7fbf05dd0640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000b1980 con 0x7fbf000b4ff0 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.517+0000 7fbf05dd0640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000b07d0 con 0x7fbf000b4670 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.517+0000 7fbf05dd0640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbf000b0ab0 con 0x7fbf0013c480 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.517+0000 7fbf05dd0640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:25.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.517+0000 7fbf05dd0640 10 monclient(hunting): authenticate will time out at 7667.621777s 2024-03-25T18:20:25.535 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.534+0000 7fbefffff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fbf0013c480 0x7fbf001e2380 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:34464/0 (socket says 172.21.15.7:34464) 2024-03-25T18:20:25.535 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.534+0000 7fbefffff640 1 -- 172.21.15.7:0/3043056173 learned_addr learned my addr 172.21.15.7:0/3043056173 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.536+0000 7fbee7fff640 10 client.?.objecter ms_handle_connect 0x7fbf000b4ff0 2024-03-25T18:20:25.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.536+0000 7fbee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.538+0000 7fbee7fff640 10 client.?.objecter ms_handle_connect 0x7fbf0013c480 2024-03-25T18:20:25.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.538+0000 7fbee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.545 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.544+0000 7fbee7fff640 10 client.?.objecter ms_handle_connect 0x7fbf000b4670 2024-03-25T18:20:25.545 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.544+0000 7fbee7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:25.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009972001 0 0) 0x7fbf000b0ab0 con 0x7fbf0013c480 2024-03-25T18:20:25.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): my global_id is 4508 2024-03-25T18:20:25.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbeec004eb0 con 0x7fbf0013c480 2024-03-25T18:20:25.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166088524 0 0) 0x7fbf000b1980 con 0x7fbf000b4ff0 2024-03-25T18:20:25.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 10 monclient(hunting): my global_id is 4488 2024-03-25T18:20:25.555 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.553+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf000b0ab0 con 0x7fbf000b4ff0 2024-03-25T18:20:25.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.559+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1877486770 0 0) 0x7fbf000b07d0 con 0x7fbf000b4670 2024-03-25T18:20:25.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.559+0000 7fbee7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:25.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.559+0000 7fbee7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:25.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.559+0000 7fbee7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:25.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.559+0000 7fbee7fff640 10 monclient(hunting): my global_id is 4489 2024-03-25T18:20:25.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.559+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbf000b1980 con 0x7fbf000b4670 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.564+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 975835298 0 0) 0x7fbeec004eb0 con 0x7fbf0013c480 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.564+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf000b07d0 con 0x7fbf0013c480 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.564+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbef8001c90 con 0x7fbf0013c480 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.564+0000 7fbee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.564+0000 7fbee7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.564+0000 7fbee7fff640 10 monclient(hunting): dump: 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:25.565 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.564+0000 7fbee7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.588+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1710937642 0 0) 0x7fbf000b0ab0 con 0x7fbf000b4ff0 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.588+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbeec004eb0 con 0x7fbf000b4ff0 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.588+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbef4001b00 con 0x7fbf000b4ff0 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.588+0000 7fbee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.588+0000 7fbee7fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.588+0000 7fbee7fff640 10 monclient(hunting): dump: 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:25.589 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:25.590 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.597+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2953866260 0 0) 0x7fbf000b1980 con 0x7fbf000b4670 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.597+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbf000b0ab0 con 0x7fbf000b4670 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.597+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbef0001830 con 0x7fbf000b4670 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.597+0000 7fbee7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.597+0000 7fbee7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.597+0000 7fbee7fff640 10 monclient(hunting): dump: 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:25.598 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:25.599 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:25.599 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1795197535 0 0) 0x7fbf000b07d0 con 0x7fbf0013c480 2024-03-25T18:20:25.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 >> v1:172.21.15.183:6789/0 conn(0x7fbf000b4670 legacy=0x7fbf000ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 >> v1:172.21.15.7:6789/0 conn(0x7fbf000b4ff0 legacy=0x7fbf000aee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:25.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:25.600 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 1 monclient: found mon.c 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf000b1460 con 0x7fbf0013c480 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbee7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 5 monclient: authenticate success, global_id 4508 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 10 monclient: _renew_subs 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf000b0ed0 con 0x7fbf0013c480 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 10 client.4508.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 10 monclient: _renew_subs 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf000af590 con 0x7fbf0013c480 2024-03-25T18:20:25.601 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.599+0000 7fbf05dd0640 1 librados: init done 2024-03-25T18:20:25.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.600+0000 7fbf05dd0640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.600+0000 7fbf05dd0640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.600+0000 7fbf05dd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:25.602 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.600+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf0006dbd0 con 0x7fbf0013c480 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.878+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbef8003990 con 0x7fbf0013c480 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.878+0000 7fbee7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.878+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbef8003d60 con 0x7fbf0013c480 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.878+0000 7fbee7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.878+0000 7fbee7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.878+0000 7fbee7fff640 10 monclient: dump: 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:25.879 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.878+0000 7fbefdffb640 4 set_mon_vals no callback set 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.879+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fbef8003f80 con 0x7fbf0013c480 2024-03-25T18:20:25.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.879+0000 7fbeff7fe640 10 monclient: get_auth_request con 0x7fbeec06e0f0 auth_method 0 2024-03-25T18:20:25.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.880+0000 7fbee7fff640 10 client.4508.objecter ms_handle_connect 0x7fbeec06e0f0 2024-03-25T18:20:25.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fbef8054b30 con 0x7fbf0013c480 2024-03-25T18:20:25.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 10 client.4508.objecter ms_dispatch 0x7fbf00000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:25.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 3 client.4508.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:25.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 3 client.4508.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:25.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 20 client.4508.objecter dump_active .. 0 homeless 2024-03-25T18:20:25.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbef8050640 con 0x7fbf0013c480 2024-03-25T18:20:25.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:25.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:25.885+0000 7fbee7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:26.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.123+0000 7fbf05dd0640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.123+0000 7fbf05dd0640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.123+0000 7fbf05dd0640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:26.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.123+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbf000688c0 con 0x7fbf0013c480 2024-03-25T18:20:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.154+0000 7fbee7fff640 1 -- 172.21.15.7:0/3043056173 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fbef8052570 con 0x7fbf0013c480 2024-03-25T18:20:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.154+0000 7fbee7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:26.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.154+0000 7fbee7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:26.156 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:26.156 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:26.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 10 librados: watch_flush enter 2024-03-25T18:20:26.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 10 librados: watch_flush exit 2024-03-25T18:20:26.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 20 client.4508.objecter shutdown clearing up homeless session... 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 10 client.4508.objecter successfully canceled tick 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 >> v1:172.21.15.7:6800/1717564767 conn(0x7fbeec06e0f0 legacy=0x7fbeec0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 10 monclient: shutdown 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 >> v1:172.21.15.7:6790/0 conn(0x7fbf0013c480 legacy=0x7fbf001e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 shutdown_connections 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 >> 172.21.15.7:0/3043056173 conn(0x7fbf000617a0 msgr2=0x7fbf001455d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 shutdown_connections 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 1 -- 172.21.15.7:0/3043056173 wait complete. 2024-03-25T18:20:26.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:26.157+0000 7fbf05dd0640 1 librados: shutdown 2024-03-25T18:20:26.170 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:32.976 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:33.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.093+0000 7f4a9ca06640 10 monclient: get_monmap_and_config 2024-03-25T18:20:33.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.093+0000 7f4a9ca06640 10 monclient: build_initial_monmap 2024-03-25T18:20:33.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.093+0000 7f4a9ca06640 10 monclient: monmap: 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:33.094648+0000 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:33.094648+0000 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:33.095 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:33.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.094+0000 7f4a9ca06640 1 Processor -- start 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.094+0000 7f4a9ca06640 1 -- start start 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.094+0000 7f4a9ca06640 10 monclient: init 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 10 monclient(hunting): picked mon.noname-a con 0x7f4a980b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 10 monclient(hunting): picked mon.noname-b con 0x7f4a980b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 10 monclient(hunting): picked mon.noname-c con 0x7f4a980b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980a96f0 con 0x7f4a980b4e70 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a980ab0e0 con 0x7f4a980b1b00 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a98145f70 con 0x7f4a980b44b0 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:33.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.095+0000 7f4a9ca06640 10 monclient(hunting): authenticate will time out at 7675.200021s 2024-03-25T18:20:33.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.103+0000 7f4a96d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4a980b1b00 0x7f4a9814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:58346/0 (socket says 172.21.15.7:58346) 2024-03-25T18:20:33.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.103+0000 7f4a96d9d640 1 -- 172.21.15.7:0/2988398302 learned_addr learned my addr 172.21.15.7:0/2988398302 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:33.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.112+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271262957 0 0) 0x7f4a980ab0e0 con 0x7f4a980b1b00 2024-03-25T18:20:33.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.112+0000 7f4a9559a640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.112+0000 7f4a9559a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.112+0000 7f4a9559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.112+0000 7f4a9559a640 10 monclient(hunting): my global_id is 4492 2024-03-25T18:20:33.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.112+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a80003520 con 0x7f4a980b1b00 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1072264909 0 0) 0x7f4a80003520 con 0x7f4a980b1b00 2024-03-25T18:20:33.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 >> v1:172.21.15.7:6790/0 conn(0x7f4a980b44b0 legacy=0x7f4a98145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 >> v1:172.21.15.7:6789/0 conn(0x7f4a980b4e70 legacy=0x7f4a9813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 10 monclient: _finish_hunting 0 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 1 monclient: found mon.noname-c 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a980a9b80 con 0x7f4a980b1b00 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 10 monclient: _finish_auth 0 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 10 monclient: _check_auth_tickets 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4a90001a10 con 0x7f4a980b1b00 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:33.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9559a640 10 monclient: dump: 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9ca06640 5 monclient: authenticate success, global_id 4492 2024-03-25T18:20:33.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.124+0000 7f4a9ca06640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a90001b70 con 0x7f4a980b1b00 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9559a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9559a640 1 -- 172.21.15.7:0/2988398302 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4a90001f80 con 0x7f4a980b1b00 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9559a640 10 monclient: dump: 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:33.130 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9ca06640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:33.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.129+0000 7f4a9ca06640 4 set_mon_vals no callback set 2024-03-25T18:20:33.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.130+0000 7f4a9ca06640 10 monclient: shutdown 2024-03-25T18:20:33.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.130+0000 7f4a9ca06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:33.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.130+0000 7f4a9ca06640 1 -- 172.21.15.7:0/2988398302 >> v1:172.21.15.183:6789/0 conn(0x7f4a980b1b00 legacy=0x7f4a9814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.130+0000 7f4a9ca06640 1 -- 172.21.15.7:0/2988398302 shutdown_connections 2024-03-25T18:20:33.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.130+0000 7f4a9ca06640 1 -- 172.21.15.7:0/2988398302 >> 172.21.15.7:0/2988398302 conn(0x7f4a980617a0 msgr2=0x7f4a98061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.130+0000 7f4a9ca06640 1 -- 172.21.15.7:0/2988398302 shutdown_connections 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.130+0000 7f4a9ca06640 1 -- 172.21.15.7:0/2988398302 wait complete. 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 10 monclient: build_initial_monmap 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 10 monclient: monmap: 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:33.134 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 1 librados: starting msgr at 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 1 librados: starting objecter 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 1 Processor -- start 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 1 -- start start 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 1 librados: setting wanted keys 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 1 librados: calling monclient init 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.131+0000 7f4a9ca06640 10 monclient: init 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 10 monclient(hunting): picked mon.noname-b con 0x7f4a980b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 10 monclient(hunting): picked mon.noname-a con 0x7f4a980b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 10 monclient(hunting): picked mon.noname-c con 0x7f4a980b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a981d4540 con 0x7f4a980b44b0 2024-03-25T18:20:33.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a98142730 con 0x7f4a980b1b00 2024-03-25T18:20:33.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4a98140160 con 0x7f4a980b4e70 2024-03-25T18:20:33.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:33.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.132+0000 7f4a9ca06640 10 monclient(hunting): authenticate will time out at 7675.236896s 2024-03-25T18:20:33.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.146+0000 7f4a9659c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f4a980b1b00 0x7f4a98140cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:58382/0 (socket says 172.21.15.7:58382) 2024-03-25T18:20:33.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.146+0000 7f4a9659c640 1 -- 172.21.15.7:0/3260184410 learned_addr learned my addr 172.21.15.7:0/3260184410 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:33.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.148+0000 7f4a86ffd640 10 client.?.objecter ms_handle_connect 0x7f4a980b1b00 2024-03-25T18:20:33.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.148+0000 7f4a86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.150+0000 7f4a86ffd640 10 client.?.objecter ms_handle_connect 0x7f4a980b44b0 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.150+0000 7f4a86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.150+0000 7f4a86ffd640 10 client.?.objecter ms_handle_connect 0x7f4a980b4e70 2024-03-25T18:20:33.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.150+0000 7f4a86ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.155+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2203461481 0 0) 0x7f4a98142730 con 0x7f4a980b1b00 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.155+0000 7f4a86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.155+0000 7f4a86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.155+0000 7f4a86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.155+0000 7f4a86ffd640 10 monclient(hunting): my global_id is 4495 2024-03-25T18:20:33.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.155+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a78004eb0 con 0x7f4a980b1b00 2024-03-25T18:20:33.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.158+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733068431 0 0) 0x7f4a981d4540 con 0x7f4a980b44b0 2024-03-25T18:20:33.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.158+0000 7f4a86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.158+0000 7f4a86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.158+0000 7f4a86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.158+0000 7f4a86ffd640 10 monclient(hunting): my global_id is 4494 2024-03-25T18:20:33.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.158+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a98142730 con 0x7f4a980b44b0 2024-03-25T18:20:33.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.169+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 607078703 0 0) 0x7f4a98142730 con 0x7f4a980b44b0 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.169+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a981d4540 con 0x7f4a980b44b0 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.169+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4a8c0018a0 con 0x7f4a980b44b0 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.169+0000 7f4a86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.169+0000 7f4a86ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.170+0000 7f4a86ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:33.171 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.170+0000 7f4a86ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.171+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 860019656 0 0) 0x7f4a78004eb0 con 0x7f4a980b1b00 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.171+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4a98142730 con 0x7f4a980b1b00 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.171+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4a88001990 con 0x7f4a980b1b00 2024-03-25T18:20:33.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.171+0000 7f4a86ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.171+0000 7f4a86ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.171+0000 7f4a86ffd640 10 monclient(hunting): dump: 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:33.173 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:33.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.173+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3123846575 0 0) 0x7f4a98140160 con 0x7f4a980b4e70 2024-03-25T18:20:33.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.173+0000 7f4a86ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:20:33.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.173+0000 7f4a86ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:33.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.173+0000 7f4a86ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:33.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.173+0000 7f4a86ffd640 10 monclient(hunting): my global_id is 4514 2024-03-25T18:20:33.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.173+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4a78004eb0 con 0x7f4a980b4e70 2024-03-25T18:20:33.178 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 984613655 0 0) 0x7f4a981d4540 con 0x7f4a980b44b0 2024-03-25T18:20:33.178 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 >> v1:172.21.15.7:6790/0 conn(0x7f4a980b4e70 legacy=0x7f4a981d50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 >> v1:172.21.15.183:6789/0 conn(0x7f4a980b1b00 legacy=0x7f4a98140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 1 monclient: found mon.a 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a981402f0 con 0x7f4a980b44b0 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 10 monclient: _finish_auth 0 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a86ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 5 monclient: authenticate success, global_id 4494 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 10 monclient: _renew_subs 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:20:33.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a981413e0 con 0x7f4a980b44b0 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 10 client.4494.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 10 monclient: _renew_subs 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a981e36e0 con 0x7f4a980b44b0 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.177+0000 7f4a9ca06640 1 librados: init done 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.178+0000 7f4a9ca06640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.180 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.178+0000 7f4a9ca06640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.178+0000 7f4a9ca06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:20:33.181 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.178+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a981d46f0 con 0x7f4a980b44b0 2024-03-25T18:20:33.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.493+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4a8c001d50 con 0x7f4a980b44b0 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.493+0000 7f4a86ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.493+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4a8c003630 con 0x7f4a980b44b0 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.493+0000 7f4a86ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.493+0000 7f4a86ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.493+0000 7f4a86ffd640 10 monclient: dump: 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:33.495 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:33.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.493+0000 7f4a94d99640 4 set_mon_vals no callback set 2024-03-25T18:20:33.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.494+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f4a8c003850 con 0x7f4a980b44b0 2024-03-25T18:20:33.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.495+0000 7f4a9659c640 10 monclient: get_auth_request con 0x7f4a7806df00 auth_method 0 2024-03-25T18:20:33.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.495+0000 7f4a86ffd640 10 client.4494.objecter ms_handle_connect 0x7f4a7806df00 2024-03-25T18:20:33.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f4a8c053340 con 0x7f4a980b44b0 2024-03-25T18:20:33.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 10 client.4494.objecter ms_dispatch 0x7f4a98000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:33.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 3 client.4494.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:33.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 3 client.4494.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:33.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 20 client.4494.objecter dump_active .. 0 homeless 2024-03-25T18:20:33.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4a8c04ec80 con 0x7f4a980b44b0 2024-03-25T18:20:33.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:33.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.503+0000 7f4a86ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:33.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.763+0000 7f4a9ca06640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.763+0000 7f4a9ca06640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.763+0000 7f4a9ca06640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:20:33.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.763+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4a9804f730 con 0x7f4a980b44b0 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.790+0000 7f4a86ffd640 1 -- 172.21.15.7:0/3260184410 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f4a8c050d80 con 0x7f4a980b44b0 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.790+0000 7f4a86ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.790+0000 7f4a86ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:33.791 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:33.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 10 librados: watch_flush enter 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 10 librados: watch_flush exit 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 20 client.4494.objecter shutdown clearing up homeless session... 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 10 client.4494.objecter successfully canceled tick 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 >> v1:172.21.15.7:6800/1717564767 conn(0x7f4a7806df00 legacy=0x7f4a780703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 10 monclient: shutdown 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 >> v1:172.21.15.7:6789/0 conn(0x7f4a980b44b0 legacy=0x7f4a981d49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 shutdown_connections 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 >> 172.21.15.7:0/3260184410 conn(0x7f4a980617a0 msgr2=0x7f4a980b6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 shutdown_connections 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 1 -- 172.21.15.7:0/3260184410 wait complete. 2024-03-25T18:20:33.794 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:33.792+0000 7f4a9ca06640 1 librados: shutdown 2024-03-25T18:20:33.804 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:34.217 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:40.854 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.232 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:40.869 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.267 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:06:40.904 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.281 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:40.918 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.287 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:40.924 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.291 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:40.917 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.295 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:40.921 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.309 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:06:40.946 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:34.319 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:06:40.946 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:40.609 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.718+0000 7f22c5e82640 10 monclient: get_monmap_and_config 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.718+0000 7f22c5e82640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.718+0000 7f22c5e82640 10 monclient: monmap: 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:40.719219+0000 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:40.719219+0000 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.719 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:40.720 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:40.720 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:40.720 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:40.720 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.718+0000 7f22c5e82640 1 Processor -- start 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 1 -- start start 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient: init 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:40.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient(hunting): picked mon.noname-b con 0x7f22c00b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient(hunting): picked mon.noname-a con 0x7f22c00b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient(hunting): picked mon.noname-c con 0x7f22c013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c00ab0e0 con 0x7f22c00b4670 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c00ab3c0 con 0x7f22c013c480 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c00a9f30 con 0x7f22c00b4ff0 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.719+0000 7f22c5e82640 10 monclient(hunting): authenticate will time out at 7682.824317s 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.720+0000 7f22beffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f22c00b4670 0x7f22c01425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:35022/0 (socket says 172.21.15.7:35022) 2024-03-25T18:20:40.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.720+0000 7f22beffd640 1 -- 172.21.15.7:0/1686319603 learned_addr learned my addr 172.21.15.7:0/1686319603 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:40.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.798+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2756539037 0 0) 0x7f22c00a9f30 con 0x7f22c00b4ff0 2024-03-25T18:20:40.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.798+0000 7f22be7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.798+0000 7f22be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.798+0000 7f22be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.799 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.798+0000 7f22be7fc640 10 monclient(hunting): my global_id is 4517 2024-03-25T18:20:40.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.798+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22a8003520 con 0x7f22c00b4ff0 2024-03-25T18:20:40.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 498045674 0 0) 0x7f22a8003520 con 0x7f22c00b4ff0 2024-03-25T18:20:40.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 >> v1:172.21.15.183:6789/0 conn(0x7f22c013c480 legacy=0x7f22c014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 >> v1:172.21.15.7:6789/0 conn(0x7f22c00b4670 legacy=0x7f22c01425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 1 monclient: found mon.noname-b 2024-03-25T18:20:40.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c00a9b70 con 0x7f22c00b4ff0 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22c5e82640 5 monclient: authenticate success, global_id 4517 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22c5e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f22b0001690 con 0x7f22c00b4ff0 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22be7fc640 10 monclient: dump: 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:40.813 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.814 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:40.814 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:40.814 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:40.814 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:40.814 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:40.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.811+0000 7f22c5e82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22b00017f0 con 0x7f22c00b4ff0 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22be7fc640 1 -- 172.21.15.7:0/1686319603 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f22b0001c00 con 0x7f22c00b4ff0 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22be7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22be7fc640 10 monclient: dump: 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.830 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:40.831 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:40.831 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:40.831 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:40.831 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:40.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22c5e82640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:40.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.829+0000 7f22c5e82640 4 set_mon_vals no callback set 2024-03-25T18:20:40.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 10 monclient: shutdown 2024-03-25T18:20:40.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:40.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 1 -- 172.21.15.7:0/1686319603 >> v1:172.21.15.7:6790/0 conn(0x7f22c00b4ff0 legacy=0x7f22c013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 1 -- 172.21.15.7:0/1686319603 shutdown_connections 2024-03-25T18:20:40.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 1 -- 172.21.15.7:0/1686319603 >> 172.21.15.7:0/1686319603 conn(0x7f22c00617a0 msgr2=0x7f22c0061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:40.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 1 -- 172.21.15.7:0/1686319603 shutdown_connections 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 1 -- 172.21.15.7:0/1686319603 wait complete. 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 10 monclient: build_initial_monmap 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.830+0000 7f22c5e82640 10 monclient: monmap: 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:40.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.831+0000 7f22c5e82640 1 librados: starting msgr at 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.831+0000 7f22c5e82640 1 librados: starting objecter 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.831+0000 7f22c5e82640 1 Processor -- start 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.831+0000 7f22c5e82640 1 -- start start 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.831+0000 7f22c5e82640 1 librados: setting wanted keys 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.831+0000 7f22c5e82640 1 librados: calling monclient init 2024-03-25T18:20:40.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.831+0000 7f22c5e82640 10 monclient: init 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 10 monclient(hunting): picked mon.noname-c con 0x7f22c00b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 10 monclient(hunting): picked mon.noname-b con 0x7f22c00b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 10 monclient(hunting): picked mon.noname-a con 0x7f22c013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c0144790 con 0x7f22c013c480 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c0144a70 con 0x7f22c00b4ff0 2024-03-25T18:20:40.835 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22c0144d50 con 0x7f22c00b4670 2024-03-25T18:20:40.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:40.836 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.832+0000 7f22c5e82640 10 monclient(hunting): authenticate will time out at 7682.937037s 2024-03-25T18:20:40.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.836+0000 7f22beffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f22c00b4ff0 0x7f22c0145ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:52364/0 (socket says 172.21.15.7:52364) 2024-03-25T18:20:40.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.836+0000 7f22beffd640 1 -- 172.21.15.7:0/2054333483 learned_addr learned my addr 172.21.15.7:0/2054333483 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:40.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.838+0000 7f22a7fff640 10 client.?.objecter ms_handle_connect 0x7f22c00b4ff0 2024-03-25T18:20:40.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.838+0000 7f22a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.847+0000 7f22a7fff640 10 client.?.objecter ms_handle_connect 0x7f22c00b4670 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.848+0000 7f22a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.848+0000 7f22a7fff640 10 client.?.objecter ms_handle_connect 0x7f22c013c480 2024-03-25T18:20:40.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.848+0000 7f22a7fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.857+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3247386968 0 0) 0x7f22c0144d50 con 0x7f22c00b4670 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.857+0000 7f22a7fff640 10 monclient(hunting): _negotiate 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.857+0000 7f22a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.857+0000 7f22a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.857+0000 7f22a7fff640 10 monclient(hunting): my global_id is 4520 2024-03-25T18:20:40.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.857+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22ac004eb0 con 0x7f22c00b4670 2024-03-25T18:20:40.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.927+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118508156 0 0) 0x7f22ac004eb0 con 0x7f22c00b4670 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.927+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f22c0144d50 con 0x7f22c00b4670 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.927+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f22b0001c00 con 0x7f22c00b4670 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.927+0000 7f22a7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.927+0000 7f22a7fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.927+0000 7f22a7fff640 10 monclient(hunting): dump: 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:40.929 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:40.930 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:40.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.927+0000 7f22a7fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:40.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1331709016 0 0) 0x7f22c0144d50 con 0x7f22c00b4670 2024-03-25T18:20:40.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 >> v1:172.21.15.183:6789/0 conn(0x7f22c00b4ff0 legacy=0x7f22c0145ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 >> v1:172.21.15.7:6789/0 conn(0x7f22c013c480 legacy=0x7f22c01e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 10 monclient: _finish_hunting 0 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 1 monclient: found mon.c 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22c0145030 con 0x7f22c00b4670 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 10 monclient: _finish_auth 0 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 10 monclient: _check_auth_tickets 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 5 monclient: authenticate success, global_id 4520 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 10 monclient: _renew_subs 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:40.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22c01435a0 con 0x7f22c00b4670 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 10 client.4520.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 10 monclient: _renew_subs 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f22c013dbe0 con 0x7f22c00b4670 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.938+0000 7f22c5e82640 1 librados: init done 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.939+0000 7f22c5e82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.939+0000 7f22c5e82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:40.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.939+0000 7f22c5e82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:40.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:40.939+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22c006dbd0 con 0x7f22c00b4670 2024-03-25T18:20:41.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.246+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f22b00038a0 con 0x7f22c00b4670 2024-03-25T18:20:41.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.246+0000 7f22a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:41.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.246+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f22b0001250 con 0x7f22c00b4670 2024-03-25T18:20:41.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.246+0000 7f22a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.246+0000 7f22a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.246+0000 7f22a7fff640 10 monclient: dump: 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:41.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.246+0000 7f22bdffb640 4 set_mon_vals no callback set 2024-03-25T18:20:41.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.247+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f22b0003d70 con 0x7f22c00b4670 2024-03-25T18:20:41.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.247+0000 7f22beffd640 10 monclient: get_auth_request con 0x7f22ac06dc70 auth_method 0 2024-03-25T18:20:41.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.248+0000 7f22a7fff640 10 client.4520.objecter ms_handle_connect 0x7f22ac06dc70 2024-03-25T18:20:41.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f22b0053790 con 0x7f22c00b4670 2024-03-25T18:20:41.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 10 client.4520.objecter ms_dispatch 0x7f22c0000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:41.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 3 client.4520.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:41.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 3 client.4520.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:41.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 20 client.4520.objecter dump_active .. 0 homeless 2024-03-25T18:20:41.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f22b004f2a0 con 0x7f22c00b4670 2024-03-25T18:20:41.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:41.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.255+0000 7f22a7fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:41.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.506+0000 7f22c5e82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:41.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.506+0000 7f22c5e82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:41.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.506+0000 7f22c5e82640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:20:41.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.506+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22c00688c0 con 0x7f22c00b4670 2024-03-25T18:20:41.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.533+0000 7f22a7fff640 1 -- 172.21.15.7:0/2054333483 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f22b00511d0 con 0x7f22c00b4670 2024-03-25T18:20:41.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.533+0000 7f22a7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:41.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.533+0000 7f22a7fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:41.534 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:41.534 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 10 librados: watch_flush enter 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 10 librados: watch_flush exit 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 20 client.4520.objecter shutdown clearing up homeless session... 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 10 client.4520.objecter successfully canceled tick 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 >> v1:172.21.15.7:6800/1717564767 conn(0x7f22ac06dc70 legacy=0x7f22ac070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 10 monclient: shutdown 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 >> v1:172.21.15.7:6790/0 conn(0x7f22c00b4670 legacy=0x7f22c01457c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 shutdown_connections 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 >> 172.21.15.7:0/2054333483 conn(0x7f22c00617a0 msgr2=0x7f22c0140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 shutdown_connections 2024-03-25T18:20:41.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 1 -- 172.21.15.7:0/2054333483 wait complete. 2024-03-25T18:20:41.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:41.535+0000 7f22c5e82640 1 librados: shutdown 2024-03-25T18:20:41.548 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:48.355 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.471+0000 7fb6914a8640 10 monclient: get_monmap_and_config 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.471+0000 7fb6914a8640 10 monclient: build_initial_monmap 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.471+0000 7fb6914a8640 10 monclient: monmap: 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:48.472251+0000 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:48.472251+0000 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:48.472 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:48.473 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:48.473 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 1 Processor -- start 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 1 -- start start 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 10 monclient: init 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb68c142e80 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb68c0af5d0 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.472+0000 7fb6914a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb68c0b0220 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:48.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.473+0000 7fb6914a8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0afb40 con 0x7fb68c0af5d0 2024-03-25T18:20:48.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.473+0000 7fb6914a8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0b1210 con 0x7fb68c142e80 2024-03-25T18:20:48.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.473+0000 7fb6914a8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c146220 con 0x7fb68c0b0220 2024-03-25T18:20:48.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.473+0000 7fb6914a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:48.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.473+0000 7fb6914a8640 10 monclient(hunting): authenticate will time out at 7690.577694s 2024-03-25T18:20:48.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.494+0000 7fb68affd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fb68c142e80 0x7fb68c06aba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:56976/0 (socket says 172.21.15.7:56976) 2024-03-25T18:20:48.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.494+0000 7fb68affd640 1 -- 172.21.15.7:0/321633 learned_addr learned my addr 172.21.15.7:0/321633 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:48.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.510+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2325420098 0 0) 0x7fb68c146220 con 0x7fb68c0b0220 2024-03-25T18:20:48.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.510+0000 7fb689ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.510+0000 7fb689ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.510+0000 7fb689ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.510+0000 7fb689ffb640 10 monclient(hunting): my global_id is 4523 2024-03-25T18:20:48.511 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.510+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb674003520 con 0x7fb68c0b0220 2024-03-25T18:20:48.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.515+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1358293584 0 0) 0x7fb68c0b1210 con 0x7fb68c142e80 2024-03-25T18:20:48.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.515+0000 7fb689ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.515+0000 7fb689ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.515+0000 7fb689ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.515+0000 7fb689ffb640 10 monclient(hunting): my global_id is 4504 2024-03-25T18:20:48.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.515+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68c146220 con 0x7fb68c142e80 2024-03-25T18:20:48.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.518+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 439082088 0 0) 0x7fb68c0afb40 con 0x7fb68c0af5d0 2024-03-25T18:20:48.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.518+0000 7fb689ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.518+0000 7fb689ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.518+0000 7fb689ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.518+0000 7fb689ffb640 10 monclient(hunting): my global_id is 4500 2024-03-25T18:20:48.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.518+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68c0b1210 con 0x7fb68c0af5d0 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2591781094 0 0) 0x7fb68c0b1210 con 0x7fb68c0af5d0 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 >> v1:172.21.15.7:6790/0 conn(0x7fb68c0b0220 legacy=0x7fb68c14dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 >> v1:172.21.15.183:6789/0 conn(0x7fb68c142e80 legacy=0x7fb68c06aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 1 monclient: found mon.noname-a 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c1463b0 con 0x7fb68c0af5d0 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 10 monclient: _finish_auth 0 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6800015b0 con 0x7fb68c0af5d0 2024-03-25T18:20:48.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.530+0000 7fb689ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.531+0000 7fb689ffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.531+0000 7fb689ffb640 10 monclient: dump: 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.531+0000 7fb6914a8640 5 monclient: authenticate success, global_id 4500 2024-03-25T18:20:48.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.531+0000 7fb6914a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:48.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb680001710 con 0x7fb68c0af5d0 2024-03-25T18:20:48.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb689ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:48.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb689ffb640 1 -- 172.21.15.7:0/321633 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb680001b20 con 0x7fb68c0af5d0 2024-03-25T18:20:48.542 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb6914a8640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb6914a8640 4 set_mon_vals no callback set 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb689ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb689ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.541+0000 7fb689ffb640 10 monclient: dump: 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:48.543 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.542+0000 7fb6914a8640 10 monclient: shutdown 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.542+0000 7fb6914a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.542+0000 7fb6914a8640 1 -- 172.21.15.7:0/321633 >> v1:172.21.15.7:6789/0 conn(0x7fb68c0af5d0 legacy=0x7fb68c145b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.542+0000 7fb6914a8640 1 -- 172.21.15.7:0/321633 shutdown_connections 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.542+0000 7fb6914a8640 1 -- 172.21.15.7:0/321633 >> 172.21.15.7:0/321633 conn(0x7fb68c0b4bd0 msgr2=0x7fb68c0b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.542+0000 7fb6914a8640 1 -- 172.21.15.7:0/321633 shutdown_connections 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.542+0000 7fb6914a8640 1 -- 172.21.15.7:0/321633 wait complete. 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.543+0000 7fb6914a8640 10 monclient: build_initial_monmap 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.543+0000 7fb6914a8640 10 monclient: monmap: 2024-03-25T18:20:48.546 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 1 librados: starting msgr at 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 1 librados: starting objecter 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 1 Processor -- start 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 1 -- start start 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 1 librados: setting wanted keys 2024-03-25T18:20:48.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 1 librados: calling monclient init 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 10 monclient: init 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 10 monclient(hunting): picked mon.noname-c con 0x7fb68c0af5d0 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 10 monclient(hunting): picked mon.noname-a con 0x7fb68c0b0220 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 10 monclient(hunting): picked mon.noname-b con 0x7fb68c142e80 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.544+0000 7fb6914a8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0713e0 con 0x7fb68c0b0220 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.545+0000 7fb6914a8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0702f0 con 0x7fb68c142e80 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.545+0000 7fb6914a8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb68c0705d0 con 0x7fb68c0af5d0 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.545+0000 7fb6914a8640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:48.548 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.545+0000 7fb6914a8640 10 monclient(hunting): authenticate will time out at 7690.649692s 2024-03-25T18:20:48.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.553+0000 7fb68a7fc640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb68c0b0220 0x7fb68c1df510 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:39978/0 (socket says 172.21.15.7:39978) 2024-03-25T18:20:48.554 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.553+0000 7fb68a7fc640 1 -- 172.21.15.7:0/108534544 learned_addr learned my addr 172.21.15.7:0/108534544 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:48.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.555+0000 7fb6737fe640 10 client.?.objecter ms_handle_connect 0x7fb68c0b0220 2024-03-25T18:20:48.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.555+0000 7fb6737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.560+0000 7fb6737fe640 10 client.?.objecter ms_handle_connect 0x7fb68c142e80 2024-03-25T18:20:48.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.560+0000 7fb6737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.570+0000 7fb6737fe640 10 client.?.objecter ms_handle_connect 0x7fb68c0af5d0 2024-03-25T18:20:48.571 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.570+0000 7fb6737fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:48.573 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.572+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3740521460 0 0) 0x7fb68c0713e0 con 0x7fb68c0b0220 2024-03-25T18:20:48.573 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.572+0000 7fb6737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.573 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.572+0000 7fb6737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.573 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.572+0000 7fb6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.573 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.572+0000 7fb6737fe640 10 monclient(hunting): my global_id is 4503 2024-03-25T18:20:48.573 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.572+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb678004eb0 con 0x7fb68c0b0220 2024-03-25T18:20:48.574 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.573+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909445365 0 0) 0x7fb68c0702f0 con 0x7fb68c142e80 2024-03-25T18:20:48.574 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.573+0000 7fb6737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.574 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.573+0000 7fb6737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.575 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.573+0000 7fb6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.575 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.573+0000 7fb6737fe640 10 monclient(hunting): my global_id is 4507 2024-03-25T18:20:48.575 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.573+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68c0713e0 con 0x7fb68c142e80 2024-03-25T18:20:48.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.584+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2705025741 0 0) 0x7fb68c0705d0 con 0x7fb68c0af5d0 2024-03-25T18:20:48.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.584+0000 7fb6737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:20:48.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.584+0000 7fb6737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:48.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.584+0000 7fb6737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:48.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.584+0000 7fb6737fe640 10 monclient(hunting): my global_id is 4526 2024-03-25T18:20:48.585 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.584+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb68c0702f0 con 0x7fb68c0af5d0 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.586+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3971180684 0 0) 0x7fb68c0713e0 con 0x7fb68c142e80 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.586+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb68c0705d0 con 0x7fb68c142e80 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.586+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb684001890 con 0x7fb68c142e80 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.586+0000 7fb6737fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.586+0000 7fb6737fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.586+0000 7fb6737fe640 10 monclient(hunting): dump: 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:48.588 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:48.589 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:48.589 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.586+0000 7fb6737fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3323776697 0 0) 0x7fb68c0705d0 con 0x7fb68c142e80 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 >> v1:172.21.15.7:6790/0 conn(0x7fb68c0af5d0 legacy=0x7fb68c070cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 >> v1:172.21.15.7:6789/0 conn(0x7fb68c0b0220 legacy=0x7fb68c1df510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 1 monclient: found mon.b 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68c0708b0 con 0x7fb68c142e80 2024-03-25T18:20:48.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 10 monclient: _finish_auth 0 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6914a8640 5 monclient: authenticate success, global_id 4507 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6914a8640 10 monclient: _renew_subs 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6914a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68c144ab0 con 0x7fb68c142e80 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.593+0000 7fb6914a8640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.594+0000 7fb6914a8640 10 monclient: _renew_subs 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.594+0000 7fb6914a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.594+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68c145590 con 0x7fb68c142e80 2024-03-25T18:20:48.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.594+0000 7fb6914a8640 1 librados: init done 2024-03-25T18:20:48.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.595+0000 7fb6914a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.595+0000 7fb6914a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.595+0000 7fb6914a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:48.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.595+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb68c142830 con 0x7fb68c142e80 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.872+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb684001ca0 con 0x7fb68c142e80 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.872+0000 7fb6737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.872+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6840037b0 con 0x7fb68c142e80 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.872+0000 7fb6737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.872+0000 7fb6737fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.872+0000 7fb6737fe640 10 monclient: dump: 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:48.873 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:48.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.872+0000 7fb6897fa640 4 set_mon_vals no callback set 2024-03-25T18:20:48.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.873+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fb684019780 con 0x7fb68c142e80 2024-03-25T18:20:48.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.873+0000 7fb68affd640 10 monclient: get_auth_request con 0x7fb67806dd80 auth_method 0 2024-03-25T18:20:48.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.873+0000 7fb6737fe640 10 client.4507.objecter ms_handle_connect 0x7fb67806dd80 2024-03-25T18:20:48.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fb684053530 con 0x7fb68c142e80 2024-03-25T18:20:48.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 10 client.4507.objecter ms_dispatch 0x7fb68c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:48.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:48.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:48.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-25T18:20:48.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb68404ee70 con 0x7fb68c142e80 2024-03-25T18:20:48.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:48.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:48.877+0000 7fb6737fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:49.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.106+0000 7fb6914a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:49.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.106+0000 7fb6914a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:49.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.106+0000 7fb6914a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:49.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.106+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb68c13fb50 con 0x7fb68c142e80 2024-03-25T18:20:49.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.139+0000 7fb6737fe640 1 -- 172.21.15.7:0/108534544 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fb684050f70 con 0x7fb68c142e80 2024-03-25T18:20:49.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.139+0000 7fb6737fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:49.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.139+0000 7fb6737fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:49.140 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:49.140 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:49.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 10 librados: watch_flush enter 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 10 librados: watch_flush exit 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 10 client.4507.objecter successfully canceled tick 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 >> v1:172.21.15.7:6800/1717564767 conn(0x7fb67806dd80 legacy=0x7fb678070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 10 monclient: shutdown 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 >> v1:172.21.15.183:6789/0 conn(0x7fb68c142e80 legacy=0x7fb68c1e2c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 shutdown_connections 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.141+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 >> 172.21.15.7:0/108534544 conn(0x7fb68c0b4bd0 msgr2=0x7fb68c074490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.142+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 shutdown_connections 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.142+0000 7fb6914a8640 1 -- 172.21.15.7:0/108534544 wait complete. 2024-03-25T18:20:49.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:49.142+0000 7fb6914a8640 1 librados: shutdown 2024-03-25T18:20:49.154 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:20:54.268 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:00.905 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.277 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:00.914 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.321 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:00.958 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.333 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:00.959 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.337 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:00.974 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.342 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:00.968 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.343 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:00.981 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.363 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:00.989 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:54.381 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:01.018 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:20:55.960 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:20:56.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.073+0000 7f3c63d9d640 10 monclient: get_monmap_and_config 2024-03-25T18:20:56.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.073+0000 7f3c63d9d640 10 monclient: build_initial_monmap 2024-03-25T18:20:56.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.073+0000 7f3c63d9d640 10 monclient: monmap: 2024-03-25T18:20:56.074 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:20:56.074779+0000 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:20:56.074779+0000 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.075 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.074+0000 7f3c63d9d640 1 Processor -- start 2024-03-25T18:20:56.076 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.074+0000 7f3c63d9d640 1 -- start start 2024-03-25T18:20:56.076 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.074+0000 7f3c63d9d640 10 monclient: init 2024-03-25T18:20:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.075+0000 7f3c63d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.075+0000 7f3c63d9d640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:20:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.075+0000 7f3c63d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c0b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.076+0000 7f3c63d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.076+0000 7f3c63d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c0b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.076+0000 7f3c63d9d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0a96f0 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.076+0000 7f3c63d9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c0ab0e0 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.076+0000 7f3c63d9d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c145f70 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.076+0000 7f3c63d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.076+0000 7f3c63d9d640 10 monclient(hunting): authenticate will time out at 7698.180773s 2024-03-25T18:20:56.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.080+0000 7f3c61b39640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f3c5c0b4e70 0x7f3c5c13f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:36306/0 (socket says 172.21.15.7:36306) 2024-03-25T18:20:56.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.080+0000 7f3c61b39640 1 -- 172.21.15.7:0/3216732727 learned_addr learned my addr 172.21.15.7:0/3216732727 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:56.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.092+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269228543 0 0) 0x7f3c5c0a96f0 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.092+0000 7f3c60b37640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.092+0000 7f3c60b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.092+0000 7f3c60b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.092+0000 7f3c60b37640 10 monclient(hunting): my global_id is 4506 2024-03-25T18:20:56.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.092+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c48003520 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.097+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 938695811 0 0) 0x7f3c5c0ab0e0 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.097+0000 7f3c60b37640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.097+0000 7f3c60b37640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.097+0000 7f3c60b37640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.097+0000 7f3c60b37640 10 monclient(hunting): my global_id is 4510 2024-03-25T18:20:56.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.097+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c0a96f0 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 360495502 0 0) 0x7f3c48003520 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 >> v1:172.21.15.7:6790/0 conn(0x7f3c5c0b44b0 legacy=0x7f3c5c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 >> v1:172.21.15.183:6789/0 conn(0x7f3c5c0b1b00 legacy=0x7f3c5c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 10 monclient: _finish_hunting 0 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 1 monclient: found mon.noname-a 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c0a9b80 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 10 monclient: _finish_auth 0 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 10 monclient: _check_auth_tickets 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3c58001690 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c60b37640 10 monclient: dump: 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.108 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.109 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:56.109 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:56.109 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:56.109 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:56.109 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c63d9d640 5 monclient: authenticate success, global_id 4506 2024-03-25T18:20:56.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.106+0000 7f3c63d9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:20:56.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.131+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c580017f0 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c60b37640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c60b37640 1 -- 172.21.15.7:0/3216732727 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3c58001c00 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c63d9d640 10 monclient: get_monmap_and_config success 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c63d9d640 4 set_mon_vals no callback set 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c60b37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c60b37640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c60b37640 10 monclient: dump: 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:56.133 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:56.134 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:56.134 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:56.134 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c63d9d640 10 monclient: shutdown 2024-03-25T18:20:56.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c63d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:56.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c63d9d640 1 -- 172.21.15.7:0/3216732727 >> v1:172.21.15.7:6789/0 conn(0x7f3c5c0b4e70 legacy=0x7f3c5c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c63d9d640 1 -- 172.21.15.7:0/3216732727 shutdown_connections 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.132+0000 7f3c63d9d640 1 -- 172.21.15.7:0/3216732727 >> 172.21.15.7:0/3216732727 conn(0x7f3c5c0617a0 msgr2=0x7f3c5c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.133+0000 7f3c63d9d640 1 -- 172.21.15.7:0/3216732727 shutdown_connections 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.133+0000 7f3c63d9d640 1 -- 172.21.15.7:0/3216732727 wait complete. 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 10 monclient: build_initial_monmap 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 10 monclient: monmap: 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:20:56.136 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 1 librados: starting msgr at 2024-03-25T18:20:56.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 1 librados: starting objecter 2024-03-25T18:20:56.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 1 Processor -- start 2024-03-25T18:20:56.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 1 -- start start 2024-03-25T18:20:56.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 1 librados: setting wanted keys 2024-03-25T18:20:56.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 1 librados: calling monclient init 2024-03-25T18:20:56.137 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.134+0000 7f3c63d9d640 10 monclient: init 2024-03-25T18:20:56.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 10 monclient: _reopen_session rank -1 2024-03-25T18:20:56.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:20:56.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 10 monclient(hunting): picked mon.noname-a con 0x7f3c5c0b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:20:56.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 10 monclient(hunting): picked mon.noname-b con 0x7f3c5c0b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:20:56.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 10 monclient(hunting): picked mon.noname-c con 0x7f3c5c0b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:20:56.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c13afa0 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.138 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c12a310 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c5c128810 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 10 monclient(hunting): _renew_subs 2024-03-25T18:20:56.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.135+0000 7f3c63d9d640 10 monclient(hunting): authenticate will time out at 7698.240076s 2024-03-25T18:20:56.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.142+0000 7f3c6233a640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f3c5c0b4e70 0x7f3c5c1e2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:58228/0 (socket says 172.21.15.7:58228) 2024-03-25T18:20:56.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.142+0000 7f3c6233a640 1 -- 172.21.15.7:0/2815148349 learned_addr learned my addr 172.21.15.7:0/2815148349 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:20:56.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.142+0000 7f3c527fc640 10 client.?.objecter ms_handle_connect 0x7f3c5c0b4e70 2024-03-25T18:20:56.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.142+0000 7f3c527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:56.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.143+0000 7f3c527fc640 10 client.?.objecter ms_handle_connect 0x7f3c5c0b44b0 2024-03-25T18:20:56.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.143+0000 7f3c527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:56.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.154+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581643902 0 0) 0x7f3c5c128810 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.154+0000 7f3c527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.154+0000 7f3c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.154+0000 7f3c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.154+0000 7f3c527fc640 10 monclient(hunting): my global_id is 4532 2024-03-25T18:20:56.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.154+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c40004eb0 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.156+0000 7f3c527fc640 10 client.?.objecter ms_handle_connect 0x7f3c5c0b1b00 2024-03-25T18:20:56.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.156+0000 7f3c527fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:20:56.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.157+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1528678623 0 0) 0x7f3c5c12a310 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.157+0000 7f3c527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.157+0000 7f3c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.157+0000 7f3c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.157+0000 7f3c527fc640 10 monclient(hunting): my global_id is 4513 2024-03-25T18:20:56.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.157+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c128810 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.170+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4020242050 0 0) 0x7f3c5c128810 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.170+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5c12a310 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.170+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3c4c0018b0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.170+0000 7f3c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.170+0000 7f3c527fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:20:56.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.170+0000 7f3c527fc640 10 monclient(hunting): dump: 2024-03-25T18:20:56.173 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.170+0000 7f3c527fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:20:56.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.174+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3718214218 0 0) 0x7f3c5c13afa0 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.174+0000 7f3c527fc640 10 monclient(hunting): _negotiate 2024-03-25T18:20:56.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.174+0000 7f3c527fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:20:56.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.174+0000 7f3c527fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:20:56.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.174+0000 7f3c527fc640 10 monclient(hunting): my global_id is 4509 2024-03-25T18:20:56.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.174+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3c5c128810 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.188+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2375399702 0 0) 0x7f3c40004eb0 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.188+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c5c13afa0 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.188+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3c54001d10 con 0x7f3c5c0b4e70 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.188+0000 7f3c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.188+0000 7f3c527fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.188+0000 7f3c527fc640 10 monclient(hunting): dump: 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:56.190 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:56.191 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.192+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3146819149 0 0) 0x7f3c5c128810 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.192+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3c40004eb0 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.192+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3c58001b00 con 0x7f3c5c0b1b00 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.192+0000 7f3c527fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.192+0000 7f3c527fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.192+0000 7f3c527fc640 10 monclient(hunting): dump: 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.193 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.194 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:56.194 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:56.194 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:56.194 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:56.194 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2529925495 0 0) 0x7f3c5c12a310 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 >> v1:172.21.15.7:6790/0 conn(0x7f3c5c0b4e70 legacy=0x7f3c5c1e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 >> v1:172.21.15.7:6789/0 conn(0x7f3c5c0b1b00 legacy=0x7f3c5c14e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 10 monclient: _finish_hunting 0 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 1 monclient: found mon.b 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c5c12a560 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 10 monclient: _finish_auth 0 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 10 monclient: _check_auth_tickets 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c527fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 5 monclient: authenticate success, global_id 4513 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 10 monclient: _renew_subs 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c12a840 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.200 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 10 client.4513.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 10 monclient: _renew_subs 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c14efc0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.198+0000 7f3c63d9d640 1 librados: init done 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.199+0000 7f3c63d9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.199+0000 7f3c63d9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.199+0000 7f3c63d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:56.201 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.199+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c1d76e0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.498+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3c4c003610 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.498+0000 7f3c527fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.498+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3c4c0038c0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.498+0000 7f3c527fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.498+0000 7f3c527fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.498+0000 7f3c527fc640 10 monclient: dump: 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:20:56.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.498+0000 7f3c53fff640 4 set_mon_vals no callback set 2024-03-25T18:20:56.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.499+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f3c4c003ae0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.500+0000 7f3c61b39640 10 monclient: get_auth_request con 0x7f3c4006e180 auth_method 0 2024-03-25T18:20:56.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.500+0000 7f3c527fc640 10 client.4513.objecter ms_handle_connect 0x7f3c4006e180 2024-03-25T18:20:56.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.504+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f3c4c0533b0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.504+0000 7f3c527fc640 10 client.4513.objecter ms_dispatch 0x7f3c5c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:20:56.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.504+0000 7f3c527fc640 3 client.4513.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:20:56.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.504+0000 7f3c527fc640 3 client.4513.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:20:56.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.504+0000 7f3c527fc640 20 client.4513.objecter dump_active .. 0 homeless 2024-03-25T18:20:56.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.505+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3c4c04eec0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.505+0000 7f3c527fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:20:56.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.505+0000 7f3c527fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:20:56.808 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.807+0000 7f3c63d9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.808 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.807+0000 7f3c63d9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.808 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.807+0000 7f3c63d9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:20:56.808 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.807+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3c5c04f730 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.837+0000 7f3c527fc640 1 -- 172.21.15.7:0/2815148349 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f3c4c050df0 con 0x7f3c5c0b44b0 2024-03-25T18:20:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.837+0000 7f3c527fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:20:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.837+0000 7f3c527fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:20:56.838 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:20:56.839 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 10 librados: watch_flush enter 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 10 librados: watch_flush exit 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 20 client.4513.objecter shutdown clearing up homeless session... 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 10 client.4513.objecter successfully canceled tick 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 >> v1:172.21.15.7:6800/1717564767 conn(0x7f3c4006e180 legacy=0x7f3c40070640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 10 monclient: shutdown 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 >> v1:172.21.15.183:6789/0 conn(0x7f3c5c0b44b0 legacy=0x7f3c5c1d79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 shutdown_connections 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 >> 172.21.15.7:0/2815148349 conn(0x7f3c5c0617a0 msgr2=0x7f3c5c0b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:20:56.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 shutdown_connections 2024-03-25T18:20:56.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 1 -- 172.21.15.7:0/2815148349 wait complete. 2024-03-25T18:20:56.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:20:56.840+0000 7f3c63d9d640 1 librados: shutdown 2024-03-25T18:20:56.852 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:03.658 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:03.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.774+0000 7eff51d23640 10 monclient: get_monmap_and_config 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.774+0000 7eff51d23640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.774+0000 7eff51d23640 10 monclient: monmap: 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:21:03.775946+0000 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:21:03.775946+0000 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:03.776 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:03.777 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:21:03.777 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:21:03.777 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.775+0000 7eff51d23640 1 Processor -- start 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.775+0000 7eff51d23640 1 -- start start 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.775+0000 7eff51d23640 10 monclient: init 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.776+0000 7eff51d23640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.776+0000 7eff51d23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.776+0000 7eff51d23640 10 monclient(hunting): picked mon.noname-b con 0x7eff4c0b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.776+0000 7eff51d23640 10 monclient(hunting): picked mon.noname-a con 0x7eff4c0b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.776+0000 7eff51d23640 10 monclient(hunting): picked mon.noname-c con 0x7eff4c13c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.776+0000 7eff51d23640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0ab0e0 con 0x7eff4c0b4670 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.776+0000 7eff51d23640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0ab3c0 con 0x7eff4c13c480 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.777+0000 7eff51d23640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0a9f30 con 0x7eff4c0b4ff0 2024-03-25T18:21:03.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.777+0000 7eff51d23640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.777+0000 7eff51d23640 10 monclient(hunting): authenticate will time out at 7705.881655s 2024-03-25T18:21:03.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.792+0000 7eff4bfff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7eff4c13c480 0x7eff4c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:48308/0 (socket says 172.21.15.7:48308) 2024-03-25T18:21:03.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.792+0000 7eff4bfff640 1 -- 172.21.15.7:0/1578687334 learned_addr learned my addr 172.21.15.7:0/1578687334 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:03.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.811+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908097353 0 0) 0x7eff4c0ab3c0 con 0x7eff4c13c480 2024-03-25T18:21:03.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.811+0000 7eff4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.811+0000 7eff4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.811+0000 7eff4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.811+0000 7eff4a7fc640 10 monclient(hunting): my global_id is 4516 2024-03-25T18:21:03.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.811+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34003520 con 0x7eff4c13c480 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1447802370 0 0) 0x7eff4c0ab0e0 con 0x7eff4c0b4670 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient(hunting): my global_id is 4512 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff4c0ab3c0 con 0x7eff4c0b4670 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2443502936 0 0) 0x7eff34003520 con 0x7eff4c13c480 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 >> v1:172.21.15.7:6790/0 conn(0x7eff4c0b4ff0 legacy=0x7eff4c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 >> v1:172.21.15.7:6789/0 conn(0x7eff4c0b4670 legacy=0x7eff4c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.829 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c0a9b70 con 0x7eff4c13c480 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7eff44001a10 con 0x7eff4c13c480 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff51d23640 5 monclient: authenticate success, global_id 4516 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff51d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff4a7fc640 10 monclient: dump: 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:03.830 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:03.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.828+0000 7eff51d23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:03.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff44001b70 con 0x7eff4c13c480 2024-03-25T18:21:03.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff4a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:03.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff4a7fc640 1 -- 172.21.15.7:0/1578687334 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7eff44001f80 con 0x7eff4c13c480 2024-03-25T18:21:03.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff51d23640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff51d23640 4 set_mon_vals no callback set 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff4a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff4a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.838+0000 7eff4a7fc640 10 monclient: dump: 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:03.840 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:03.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.839+0000 7eff51d23640 10 monclient: shutdown 2024-03-25T18:21:03.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.839+0000 7eff51d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:03.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.839+0000 7eff51d23640 1 -- 172.21.15.7:0/1578687334 >> v1:172.21.15.183:6789/0 conn(0x7eff4c13c480 legacy=0x7eff4c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.839+0000 7eff51d23640 1 -- 172.21.15.7:0/1578687334 shutdown_connections 2024-03-25T18:21:03.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.839+0000 7eff51d23640 1 -- 172.21.15.7:0/1578687334 >> 172.21.15.7:0/1578687334 conn(0x7eff4c0617a0 msgr2=0x7eff4c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:03.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.839+0000 7eff51d23640 1 -- 172.21.15.7:0/1578687334 shutdown_connections 2024-03-25T18:21:03.842 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.839+0000 7eff51d23640 1 -- 172.21.15.7:0/1578687334 wait complete. 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.840+0000 7eff51d23640 10 monclient: build_initial_monmap 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.840+0000 7eff51d23640 10 monclient: monmap: 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:03.843 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.841+0000 7eff51d23640 1 librados: starting msgr at 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.841+0000 7eff51d23640 1 librados: starting objecter 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.841+0000 7eff51d23640 1 Processor -- start 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.841+0000 7eff51d23640 1 -- start start 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.841+0000 7eff51d23640 1 librados: setting wanted keys 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.841+0000 7eff51d23640 1 librados: calling monclient init 2024-03-25T18:21:03.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.841+0000 7eff51d23640 10 monclient: init 2024-03-25T18:21:03.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:03.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:03.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 10 monclient(hunting): picked mon.noname-b con 0x7eff4c0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:03.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 10 monclient(hunting): picked mon.noname-a con 0x7eff4c0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:03.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 10 monclient(hunting): picked mon.noname-c con 0x7eff4c13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:03.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0b03d0 con 0x7eff4c0b4ff0 2024-03-25T18:21:03.845 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0b06b0 con 0x7eff4c0b4670 2024-03-25T18:21:03.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff4c0ae4b0 con 0x7eff4c13c480 2024-03-25T18:21:03.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:03.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.842+0000 7eff51d23640 10 monclient(hunting): authenticate will time out at 7705.947097s 2024-03-25T18:21:03.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.847+0000 7eff4b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7eff4c0b4670 0x7eff4c0b1120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:48336/0 (socket says 172.21.15.7:48336) 2024-03-25T18:21:03.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.847+0000 7eff4b7fe640 1 -- 172.21.15.7:0/779645611 learned_addr learned my addr 172.21.15.7:0/779645611 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:03.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.848+0000 7eff33fff640 10 client.?.objecter ms_handle_connect 0x7eff4c0b4670 2024-03-25T18:21:03.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.848+0000 7eff33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.854+0000 7eff33fff640 10 client.?.objecter ms_handle_connect 0x7eff4c13c480 2024-03-25T18:21:03.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.854+0000 7eff33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.864+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137333828 0 0) 0x7eff4c0b06b0 con 0x7eff4c0b4670 2024-03-25T18:21:03.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.864+0000 7eff33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.864+0000 7eff33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.864+0000 7eff33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.864+0000 7eff33fff640 10 monclient(hunting): my global_id is 4519 2024-03-25T18:21:03.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.864+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff38004eb0 con 0x7eff4c0b4670 2024-03-25T18:21:03.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.868+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1833525766 0 0) 0x7eff4c0ae4b0 con 0x7eff4c13c480 2024-03-25T18:21:03.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.868+0000 7eff33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.868+0000 7eff33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.868+0000 7eff33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.868+0000 7eff33fff640 10 monclient(hunting): my global_id is 4538 2024-03-25T18:21:03.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.868+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff4c0b06b0 con 0x7eff4c13c480 2024-03-25T18:21:03.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.871+0000 7eff33fff640 10 client.?.objecter ms_handle_connect 0x7eff4c0b4ff0 2024-03-25T18:21:03.872 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.871+0000 7eff33fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:03.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.887+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136876185 0 0) 0x7eff4c0b03d0 con 0x7eff4c0b4ff0 2024-03-25T18:21:03.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.887+0000 7eff33fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:03.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.887+0000 7eff33fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:03.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.887+0000 7eff33fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:03.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.887+0000 7eff33fff640 10 monclient(hunting): my global_id is 4515 2024-03-25T18:21:03.888 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.887+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff4c0ae4b0 con 0x7eff4c0b4ff0 2024-03-25T18:21:03.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.889+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 889230566 0 0) 0x7eff38004eb0 con 0x7eff4c0b4670 2024-03-25T18:21:03.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.889+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff4c0b03d0 con 0x7eff4c0b4670 2024-03-25T18:21:03.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.889+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7eff3c001a30 con 0x7eff4c0b4670 2024-03-25T18:21:03.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.889+0000 7eff33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.889+0000 7eff33fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:03.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.889+0000 7eff33fff640 10 monclient(hunting): dump: 2024-03-25T18:21:03.890 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:03.891 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.889+0000 7eff33fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:03.904 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.903+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2289401991 0 0) 0x7eff4c0b06b0 con 0x7eff4c13c480 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.903+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7eff38004eb0 con 0x7eff4c13c480 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.903+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7eff44001e80 con 0x7eff4c13c480 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.903+0000 7eff33fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.903+0000 7eff33fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.903+0000 7eff33fff640 10 monclient(hunting): dump: 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:03.905 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:03.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3088967313 0 0) 0x7eff4c0b03d0 con 0x7eff4c0b4670 2024-03-25T18:21:03.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 >> v1:172.21.15.7:6790/0 conn(0x7eff4c13c480 legacy=0x7eff4c0af0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 >> v1:172.21.15.7:6789/0 conn(0x7eff4c0b4ff0 legacy=0x7eff4c0ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:03.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:03.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 1 monclient: found mon.b 2024-03-25T18:21:03.908 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff4c0b1830 con 0x7eff4c0b4670 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff33fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff51d23640 5 monclient: authenticate success, global_id 4519 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff51d23640 10 monclient: _renew_subs 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff51d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.906+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff4c1e3610 con 0x7eff4c0b4670 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.907+0000 7eff51d23640 10 client.4519.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.907+0000 7eff51d23640 10 monclient: _renew_subs 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.907+0000 7eff51d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.907+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7eff4c1e3980 con 0x7eff4c0b4670 2024-03-25T18:21:03.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.907+0000 7eff51d23640 1 librados: init done 2024-03-25T18:21:03.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.908+0000 7eff51d23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.908+0000 7eff51d23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:03.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.908+0000 7eff51d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:03.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:03.908+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff4c0ae640 con 0x7eff4c0b4670 2024-03-25T18:21:04.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.170+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7eff3c001c70 con 0x7eff4c0b4670 2024-03-25T18:21:04.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.170+0000 7eff33fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:04.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.170+0000 7eff49ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.188+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7eff3c003a60 con 0x7eff4c0b4670 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.188+0000 7eff33fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.188+0000 7eff33fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.188+0000 7eff33fff640 10 monclient: dump: 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:04.189 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:04.190 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:04.190 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:04.190 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:04.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.189+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7eff3c003c40 con 0x7eff4c0b4670 2024-03-25T18:21:04.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.189+0000 7eff4affd640 10 monclient: get_auth_request con 0x7eff3806df40 auth_method 0 2024-03-25T18:21:04.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.190+0000 7eff33fff640 10 client.4519.objecter ms_handle_connect 0x7eff3806df40 2024-03-25T18:21:04.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7eff3c054950 con 0x7eff4c0b4670 2024-03-25T18:21:04.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 10 client.4519.objecter ms_dispatch 0x7eff4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:04.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 3 client.4519.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:04.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 3 client.4519.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:04.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 20 client.4519.objecter dump_active .. 0 homeless 2024-03-25T18:21:04.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7eff3c050460 con 0x7eff4c0b4670 2024-03-25T18:21:04.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:04.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.195+0000 7eff33fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:04.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.453+0000 7eff51d23640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.453+0000 7eff51d23640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.453+0000 7eff51d23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:04.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.453+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7eff4c06dbd0 con 0x7eff4c0b4670 2024-03-25T18:21:04.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.499+0000 7eff33fff640 1 -- 172.21.15.7:0/779645611 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7eff3c052390 con 0x7eff4c0b4670 2024-03-25T18:21:04.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.499+0000 7eff33fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:04.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.499+0000 7eff33fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:04.501 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:21:04.501 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:04.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.501+0000 7eff51d23640 10 librados: watch_flush enter 2024-03-25T18:21:04.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.501+0000 7eff51d23640 10 librados: watch_flush exit 2024-03-25T18:21:04.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.501+0000 7eff51d23640 20 client.4519.objecter shutdown clearing up homeless session... 2024-03-25T18:21:04.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.501+0000 7eff51d23640 10 client.4519.objecter successfully canceled tick 2024-03-25T18:21:04.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.501+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 >> v1:172.21.15.7:6800/1717564767 conn(0x7eff3806df40 legacy=0x7eff38070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 10 monclient: shutdown 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 >> v1:172.21.15.183:6789/0 conn(0x7eff4c0b4670 legacy=0x7eff4c0b1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 shutdown_connections 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 >> 172.21.15.7:0/779645611 conn(0x7eff4c0617a0 msgr2=0x7eff4c145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 shutdown_connections 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 1 -- 172.21.15.7:0/779645611 wait complete. 2024-03-25T18:21:04.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:04.502+0000 7eff51d23640 1 librados: shutdown 2024-03-25T18:21:04.514 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:11.320 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:11.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.424+0000 7f4ec1b56640 10 monclient: get_monmap_and_config 2024-03-25T18:21:11.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.424+0000 7f4ec1b56640 10 monclient: build_initial_monmap 2024-03-25T18:21:11.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.424+0000 7f4ec1b56640 10 monclient: monmap: 2024-03-25T18:21:11.425 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:11.425 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:11.425 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:21:11.425495+0000 2024-03-25T18:21:11.425 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:21:11.425495+0000 2024-03-25T18:21:11.426 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:11.426 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:11.426 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:11.426 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:21:11.426 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:21:11.426 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 1 Processor -- start 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 1 -- start start 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 10 monclient: init 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 10 monclient(hunting): picked mon.noname-a con 0x7f4ebc0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:11.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 10 monclient(hunting): picked mon.noname-b con 0x7f4ebc0b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 10 monclient(hunting): picked mon.noname-c con 0x7f4ebc13c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.425+0000 7f4ec1b56640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0ab0e0 con 0x7f4ebc0b4ff0 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.426+0000 7f4ec1b56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0ab3c0 con 0x7f4ebc13c480 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.426+0000 7f4ec1b56640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc0a9f30 con 0x7f4ebc0b4670 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.426+0000 7f4ec1b56640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.426+0000 7f4ec1b56640 10 monclient(hunting): authenticate will time out at 7713.530664s 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.426+0000 7f4ebb7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4ebc0b4ff0 0x7f4ebc13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55938/0 (socket says 172.21.15.7:55938) 2024-03-25T18:21:11.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.426+0000 7f4ebb7fe640 1 -- 172.21.15.7:0/3045318425 learned_addr learned my addr 172.21.15.7:0/3045318425 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:11.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.452+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1410302824 0 0) 0x7f4ebc0a9f30 con 0x7f4ebc0b4670 2024-03-25T18:21:11.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.452+0000 7f4eba7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.452+0000 7f4eba7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.452+0000 7f4eba7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.452+0000 7f4eba7fc640 10 monclient(hunting): my global_id is 4541 2024-03-25T18:21:11.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.453+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea4003520 con 0x7f4ebc0b4670 2024-03-25T18:21:11.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 189792243 0 0) 0x7f4ea4003520 con 0x7f4ebc0b4670 2024-03-25T18:21:11.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 >> v1:172.21.15.183:6789/0 conn(0x7f4ebc13c480 legacy=0x7f4ebc14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:21:11.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 >> v1:172.21.15.7:6789/0 conn(0x7f4ebc0b4ff0 legacy=0x7f4ebc13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 1 monclient: found mon.noname-b 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc0a9b70 con 0x7f4ebc0b4670 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 10 monclient: _finish_auth 0 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4eb0001690 con 0x7f4ebc0b4670 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4ec1b56640 5 monclient: authenticate success, global_id 4541 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4eba7fc640 10 monclient: dump: 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.468 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.469 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.469 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:11.469 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:11.469 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:11.469 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:11.469 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:11.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.466+0000 7f4ec1b56640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:11.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4eb0001860 con 0x7f4ebc0b4670 2024-03-25T18:21:11.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4eba7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:11.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4eba7fc640 1 -- 172.21.15.7:0/3045318425 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4eb0001cb0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4ec1b56640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4ec1b56640 4 set_mon_vals no callback set 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4eba7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4eba7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.475+0000 7f4eba7fc640 10 monclient: dump: 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:11.477 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.476+0000 7f4ec1b56640 10 monclient: shutdown 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.476+0000 7f4ec1b56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.476+0000 7f4ec1b56640 1 -- 172.21.15.7:0/3045318425 >> v1:172.21.15.7:6790/0 conn(0x7f4ebc0b4670 legacy=0x7f4ebc1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.476+0000 7f4ec1b56640 1 -- 172.21.15.7:0/3045318425 shutdown_connections 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.476+0000 7f4ec1b56640 1 -- 172.21.15.7:0/3045318425 >> 172.21.15.7:0/3045318425 conn(0x7f4ebc0617a0 msgr2=0x7f4ebc061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.476+0000 7f4ec1b56640 1 -- 172.21.15.7:0/3045318425 shutdown_connections 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.477+0000 7f4ec1b56640 1 -- 172.21.15.7:0/3045318425 wait complete. 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 10 monclient: build_initial_monmap 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 10 monclient: monmap: 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:11.479 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:21:11.480 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:11.480 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:21:11.480 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:21:11.480 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:11.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 1 librados: starting msgr at 2024-03-25T18:21:11.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 1 librados: starting objecter 2024-03-25T18:21:11.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 1 Processor -- start 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 1 -- start start 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 1 librados: setting wanted keys 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 1 librados: calling monclient init 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.478+0000 7f4ec1b56640 10 monclient: init 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 10 monclient(hunting): picked mon.noname-c con 0x7f4ebc0b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:11.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 10 monclient(hunting): picked mon.noname-a con 0x7f4ebc0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 10 monclient(hunting): picked mon.noname-b con 0x7f4ebc13c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc146350 con 0x7f4ebc0b4ff0 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc144490 con 0x7f4ebc13c480 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ebc144770 con 0x7f4ebc0b4670 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.479+0000 7f4ec1b56640 10 monclient(hunting): authenticate will time out at 7713.583974s 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.480+0000 7f4ebaffd640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f4ebc0b4ff0 0x7f4ebc1df150 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55966/0 (socket says 172.21.15.7:55966) 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.480+0000 7f4ebaffd640 1 -- 172.21.15.7:0/1425399903 learned_addr learned my addr 172.21.15.7:0/1425399903 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.481+0000 7f4ea3fff640 10 client.?.objecter ms_handle_connect 0x7f4ebc0b4ff0 2024-03-25T18:21:11.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.481+0000 7f4ea3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.501+0000 7f4ea3fff640 10 client.?.objecter ms_handle_connect 0x7f4ebc0b4670 2024-03-25T18:21:11.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.501+0000 7f4ea3fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:11.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.506+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1138397033 0 0) 0x7f4ebc144770 con 0x7f4ebc0b4670 2024-03-25T18:21:11.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.506+0000 7f4ea3fff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:11.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.506+0000 7f4ea3fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:11.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.506+0000 7f4ea3fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:11.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.506+0000 7f4ea3fff640 10 monclient(hunting): my global_id is 4544 2024-03-25T18:21:11.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.506+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4ea8004eb0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.524 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.523+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3118384405 0 0) 0x7f4ea8004eb0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.523+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4ebc144770 con 0x7f4ebc0b4670 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.523+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4eac0018a0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.523+0000 7f4ea3fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.523+0000 7f4ea3fff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.523+0000 7f4ea3fff640 10 monclient(hunting): dump: 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:11.525 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:11.526 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:11.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.523+0000 7f4ea3fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 439306418 0 0) 0x7f4ebc144770 con 0x7f4ebc0b4670 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 >> v1:172.21.15.183:6789/0 conn(0x7f4ebc13c480 legacy=0x7f4ebc1e2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 >> v1:172.21.15.7:6789/0 conn(0x7f4ebc0b4ff0 legacy=0x7f4ebc1df150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 1 monclient: found mon.c 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ebc144a50 con 0x7f4ebc0b4670 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 10 monclient: _finish_auth 0 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:11.536 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ea3fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 5 monclient: authenticate success, global_id 4544 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 10 monclient: _renew_subs 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ebc143940 con 0x7f4ebc0b4670 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 10 client.4544.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 10 monclient: _renew_subs 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ebc1340c0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.534+0000 7f4ec1b56640 1 librados: init done 2024-03-25T18:21:11.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.535+0000 7f4ec1b56640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.535+0000 7f4ec1b56640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.535+0000 7f4ec1b56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:11.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.535+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ebc143790 con 0x7f4ebc0b4670 2024-03-25T18:21:11.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.804+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4eac0047e0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.804+0000 7f4ea3fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.804+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4eac004bf0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.804+0000 7f4ea3fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.804+0000 7f4ea3fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.804+0000 7f4ea3fff640 10 monclient: dump: 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:11.806 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:11.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.804+0000 7f4eb9ffb640 4 set_mon_vals no callback set 2024-03-25T18:21:11.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.805+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f4eac004e10 con 0x7f4ebc0b4670 2024-03-25T18:21:11.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.806+0000 7f4ebaffd640 10 monclient: get_auth_request con 0x7f4ea806dc70 auth_method 0 2024-03-25T18:21:11.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.806+0000 7f4ea3fff640 10 client.4544.objecter ms_handle_connect 0x7f4ea806dc70 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f4eac0547f0 con 0x7f4ebc0b4670 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 10 client.4544.objecter ms_dispatch 0x7f4ebc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 3 client.4544.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 3 client.4544.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 20 client.4544.objecter dump_active .. 0 homeless 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f4eac050300 con 0x7f4ebc0b4670 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:11.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:11.812+0000 7f4ea3fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:12.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.043+0000 7f4ec1b56640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:12.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.043+0000 7f4ec1b56640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:12.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.043+0000 7f4ec1b56640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:12.045 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.043+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4ebc06dbd0 con 0x7f4ebc0b4670 2024-03-25T18:21:12.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.073+0000 7f4ea3fff640 1 -- 172.21.15.7:0/1425399903 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f4eac052230 con 0x7f4ebc0b4670 2024-03-25T18:21:12.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.073+0000 7f4ea3fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:12.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.073+0000 7f4ea3fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:12.075 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:21:12.075 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 10 librados: watch_flush enter 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 10 librados: watch_flush exit 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 20 client.4544.objecter shutdown clearing up homeless session... 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 10 client.4544.objecter successfully canceled tick 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 >> v1:172.21.15.7:6800/1717564767 conn(0x7f4ea806dc70 legacy=0x7f4ea8070130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 10 monclient: shutdown 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 >> v1:172.21.15.7:6790/0 conn(0x7f4ebc0b4670 legacy=0x7f4ebc144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 shutdown_connections 2024-03-25T18:21:12.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 >> 172.21.15.7:0/1425399903 conn(0x7f4ebc0617a0 msgr2=0x7f4ebc0b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:12.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 shutdown_connections 2024-03-25T18:21:12.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.075+0000 7f4ec1b56640 1 -- 172.21.15.7:0/1425399903 wait complete. 2024-03-25T18:21:12.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:12.076+0000 7f4ec1b56640 1 librados: shutdown 2024-03-25T18:21:12.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:18.894 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:18.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.994+0000 7f11e09cf640 10 monclient: get_monmap_and_config 2024-03-25T18:21:18.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.994+0000 7f11e09cf640 10 monclient: build_initial_monmap 2024-03-25T18:21:18.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.994+0000 7f11e09cf640 10 monclient: monmap: 2024-03-25T18:21:18.995 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:18.995 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:21:18.995489+0000 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:21:18.995489+0000 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:21:18.996 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:18.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 1 Processor -- start 2024-03-25T18:21:18.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 1 -- start start 2024-03-25T18:21:18.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient: init 2024-03-25T18:21:18.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:18.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:18.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient(hunting): picked mon.noname-b con 0x7f11dc0b49f0 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:18.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient(hunting): picked mon.noname-c con 0x7f11dc0b4dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient(hunting): picked mon.noname-a con 0x7f11dc0a9a90 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0aa070 con 0x7f11dc0a9a90 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0aaf40 con 0x7f11dc0b4dd0 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0ab220 con 0x7f11dc0b49f0 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.995+0000 7f11e09cf640 10 monclient(hunting): authenticate will time out at 7721.100546s 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.996+0000 7f11dad9d640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f11dc0a9a90 0x7f11dc142c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:57526/0 (socket says 172.21.15.7:57526) 2024-03-25T18:21:18.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:18.996+0000 7f11dad9d640 1 -- 172.21.15.7:0/3497713636 learned_addr learned my addr 172.21.15.7:0/3497713636 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:21.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient(hunting): tick 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient(hunting): _check_auth_tickets 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 1 monclient(hunting): continuing hunt 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient(hunting): _reopen_session rank -1 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 1 -- 172.21.15.7:0/3497713636 >> v1:172.21.15.7:6790/0 conn(0x7f11dc0b49f0 legacy=0x7f11dc13bd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 1 -- 172.21.15.7:0/3497713636 >> v1:172.21.15.183:6789/0 conn(0x7f11dc0b4dd0 legacy=0x7f11dc142520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 1 -- 172.21.15.7:0/3497713636 >> v1:172.21.15.7:6789/0 conn(0x7f11dc0a9a90 legacy=0x7f11dc142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient(hunting): picked mon.noname-b con 0x7f11bc0036e0 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient(hunting): picked mon.noname-c con 0x7f11dc0b4dd0 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:21.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient(hunting): picked mon.noname-a con 0x7f11dc0a9a90 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:21.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 1 -- 172.21.15.7:0/3497713636 --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc13c410 con 0x7f11dc0a9a90 2024-03-25T18:21:21.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 1 -- 172.21.15.7:0/3497713636 --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc00f3f0 con 0x7f11dc0b4dd0 2024-03-25T18:21:21.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 1 -- 172.21.15.7:0/3497713636 --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11bc010650 con 0x7f11bc0036e0 2024-03-25T18:21:21.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:21.996+0000 7f11d8d99640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:24.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.845+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3934350444 0 0) 0x7f11bc010650 con 0x7f11bc0036e0 2024-03-25T18:21:24.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.845+0000 7f11d9d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:24.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.845+0000 7f11d9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:24.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.845+0000 7f11d9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:24.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.845+0000 7f11d9d9b640 10 monclient(hunting): my global_id is 4547 2024-03-25T18:21:24.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.845+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11c8003520 con 0x7f11bc0036e0 2024-03-25T18:21:24.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3512430105 0 0) 0x7f11c8003520 con 0x7f11bc0036e0 2024-03-25T18:21:24.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 >> v1:172.21.15.183:6789/0 conn(0x7f11dc0b4dd0 legacy=0x7f11bc0093d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 >> v1:172.21.15.7:6789/0 conn(0x7f11dc0a9a90 legacy=0x7f11bc00cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:24.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 1 monclient: found mon.noname-b 2024-03-25T18:21:24.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:24.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11bc0118b0 con 0x7f11bc0036e0 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 10 monclient: _finish_auth 0 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.865+0000 7f11d9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.866+0000 7f11e09cf640 5 monclient: authenticate success, global_id 4547 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.866+0000 7f11e09cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.866+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11cc0016d0 con 0x7f11bc0036e0 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.866+0000 7f11d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.866+0000 7f11d9d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.866+0000 7f11d9d9b640 10 monclient: dump: 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:24.868 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:24.869 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:24.869 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:24.869 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:24.869 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:24.869 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:24.869 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:24.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.866+0000 7f11e09cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:24.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11cc0018a0 con 0x7f11bc0036e0 2024-03-25T18:21:24.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11d9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:24.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11d9d9b640 1 -- 172.21.15.7:0/3497713636 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11cc001c30 con 0x7f11bc0036e0 2024-03-25T18:21:24.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11d9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11d9d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11d9d9b640 10 monclient: dump: 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11e09cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:24.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.873+0000 7f11e09cf640 4 set_mon_vals no callback set 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.874+0000 7f11e09cf640 10 monclient: shutdown 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.874+0000 7f11e09cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.874+0000 7f11e09cf640 1 -- 172.21.15.7:0/3497713636 >> v1:172.21.15.7:6790/0 conn(0x7f11bc0036e0 legacy=0x7f11bc005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.874+0000 7f11e09cf640 1 -- 172.21.15.7:0/3497713636 shutdown_connections 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.874+0000 7f11e09cf640 1 -- 172.21.15.7:0/3497713636 >> 172.21.15.7:0/3497713636 conn(0x7f11dc0615c0 msgr2=0x7f11dc0619a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.874+0000 7f11e09cf640 1 -- 172.21.15.7:0/3497713636 shutdown_connections 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.874+0000 7f11e09cf640 1 -- 172.21.15.7:0/3497713636 wait complete. 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.875+0000 7f11e09cf640 10 monclient: build_initial_monmap 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.875+0000 7f11e09cf640 10 monclient: monmap: 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:24.877 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.876+0000 7f11e09cf640 1 librados: starting msgr at 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.876+0000 7f11e09cf640 1 librados: starting objecter 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.876+0000 7f11e09cf640 1 Processor -- start 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.876+0000 7f11e09cf640 1 -- start start 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.876+0000 7f11e09cf640 1 librados: setting wanted keys 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.876+0000 7f11e09cf640 1 librados: calling monclient init 2024-03-25T18:21:24.878 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.876+0000 7f11e09cf640 10 monclient: init 2024-03-25T18:21:24.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 10 monclient(hunting): picked mon.noname-c con 0x7f11bc0036e0 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 10 monclient(hunting): picked mon.noname-b con 0x7f11dc0a9a90 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 10 monclient(hunting): picked mon.noname-a con 0x7f11dc0b4dd0 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc1db260 con 0x7f11dc0b4dd0 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc0ac830 con 0x7f11dc0a9a90 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11dc1db6f0 con 0x7f11bc0036e0 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:24.880 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.877+0000 7f11e09cf640 10 monclient(hunting): authenticate will time out at 7726.981888s 2024-03-25T18:21:24.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.880+0000 7f11d3fff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f11dc0a9a90 0x7f11dc13ebe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:42708/0 (socket says 172.21.15.7:42708) 2024-03-25T18:21:24.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.881+0000 7f11d3fff640 1 -- 172.21.15.7:0/2785858940 learned_addr learned my addr 172.21.15.7:0/2785858940 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:24.882 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.881+0000 7f11d2ffd640 10 client.?.objecter ms_handle_connect 0x7f11dc0a9a90 2024-03-25T18:21:24.882 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.881+0000 7f11d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:24.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.893+0000 7f11d2ffd640 10 client.?.objecter ms_handle_connect 0x7f11bc0036e0 2024-03-25T18:21:24.894 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.893+0000 7f11d2ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:24.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.900+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 438798221 0 0) 0x7f11dc1db6f0 con 0x7f11bc0036e0 2024-03-25T18:21:24.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.900+0000 7f11d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:24.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.900+0000 7f11d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:24.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.900+0000 7f11d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:24.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.900+0000 7f11d2ffd640 10 monclient(hunting): my global_id is 4550 2024-03-25T18:21:24.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.900+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11bc006aa0 con 0x7f11bc0036e0 2024-03-25T18:21:24.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.912+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3964881344 0 0) 0x7f11bc006aa0 con 0x7f11bc0036e0 2024-03-25T18:21:24.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.912+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11dc1db6f0 con 0x7f11bc0036e0 2024-03-25T18:21:24.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.912+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11cc001bb0 con 0x7f11bc0036e0 2024-03-25T18:21:24.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.912+0000 7f11d2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:24.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.912+0000 7f11d2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.912+0000 7f11d2ffd640 10 monclient(hunting): dump: 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:24.914 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.912+0000 7f11d2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1778935119 0 0) 0x7f11dc1db6f0 con 0x7f11bc0036e0 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 >> v1:172.21.15.183:6789/0 conn(0x7f11dc0a9a90 legacy=0x7f11dc13ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 >> v1:172.21.15.7:6789/0 conn(0x7f11dc0b4dd0 legacy=0x7f11dc1e2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 1 monclient: found mon.c 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11dc13b950 con 0x7f11bc0036e0 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 5 monclient: authenticate success, global_id 4550 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 10 monclient: _renew_subs 2024-03-25T18:21:24.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11dc0b1680 con 0x7f11bc0036e0 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 10 client.4550.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 10 monclient: _renew_subs 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11dc13f4d0 con 0x7f11bc0036e0 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.933+0000 7f11e09cf640 1 librados: init done 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.934+0000 7f11e09cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:24.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.934+0000 7f11e09cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:24.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.934+0000 7f11e09cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:24.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:24.934+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11dc04f730 con 0x7f11bc0036e0 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.215+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11cc001d10 con 0x7f11bc0036e0 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.215+0000 7f11d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.215+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f11cc003a40 con 0x7f11bc0036e0 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.215+0000 7f11d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.215+0000 7f11d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.215+0000 7f11d2ffd640 10 monclient: dump: 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:25.217 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:25.218 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.215+0000 7f11d959a640 4 set_mon_vals no callback set 2024-03-25T18:21:25.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.217+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f11cc019ad0 con 0x7f11bc0036e0 2024-03-25T18:21:25.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.217+0000 7f11d3fff640 10 monclient: get_auth_request con 0x7f11bc06e610 auth_method 0 2024-03-25T18:21:25.219 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.217+0000 7f11d2ffd640 10 client.4550.objecter ms_handle_connect 0x7f11bc06e610 2024-03-25T18:21:25.224 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.223+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f11cc0536f0 con 0x7f11bc0036e0 2024-03-25T18:21:25.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.223+0000 7f11d2ffd640 10 client.4550.objecter ms_dispatch 0x7f11dc000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:25.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.223+0000 7f11d2ffd640 3 client.4550.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:25.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.223+0000 7f11d2ffd640 3 client.4550.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:25.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.224+0000 7f11d2ffd640 20 client.4550.objecter dump_active .. 0 homeless 2024-03-25T18:21:25.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.224+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f11cc04f030 con 0x7f11bc0036e0 2024-03-25T18:21:25.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.224+0000 7f11d2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:25.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.224+0000 7f11d2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:25.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.440+0000 7f11e09cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:25.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.440+0000 7f11e09cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:25.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.440+0000 7f11e09cf640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:21:25.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.440+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f11dc0b6b90 con 0x7f11bc0036e0 2024-03-25T18:21:25.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.499+0000 7f11d2ffd640 1 -- 172.21.15.7:0/2785858940 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f11cc051130 con 0x7f11bc0036e0 2024-03-25T18:21:25.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.499+0000 7f11d2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:25.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.499+0000 7f11d2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:25.501 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:21:25.501 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:25.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 10 librados: watch_flush enter 2024-03-25T18:21:25.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 10 librados: watch_flush exit 2024-03-25T18:21:25.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 20 client.4550.objecter shutdown clearing up homeless session... 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 10 client.4550.objecter successfully canceled tick 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 >> v1:172.21.15.7:6800/1717564767 conn(0x7f11bc06e610 legacy=0x7f11bc070ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 10 monclient: shutdown 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 >> v1:172.21.15.7:6790/0 conn(0x7f11bc0036e0 legacy=0x7f11dc13b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 shutdown_connections 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 >> 172.21.15.7:0/2785858940 conn(0x7f11dc0615c0 msgr2=0x7f11dc142c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.501+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 shutdown_connections 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.502+0000 7f11e09cf640 1 -- 172.21.15.7:0/2785858940 wait complete. 2024-03-25T18:21:25.504 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:25.502+0000 7f11e09cf640 1 librados: shutdown 2024-03-25T18:21:25.515 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:28.764 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:35.401 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.772 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:35.409 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.821 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:35.458 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.828 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:35.454 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.833 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:35.470 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.840 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:35.478 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.842 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:35.468 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.864 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:35.491 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:28.869 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:35.506 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:32.322 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.434+0000 7f012d13e640 10 monclient: get_monmap_and_config 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.434+0000 7f012d13e640 10 monclient: build_initial_monmap 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.434+0000 7f012d13e640 10 monclient: monmap: 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:21:32.435484+0000 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:21:32.435484+0000 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:32.435 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:32.436 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:32.436 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:21:32.436 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:21:32.436 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.435+0000 7f012d13e640 1 Processor -- start 2024-03-25T18:21:32.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.435+0000 7f012d13e640 1 -- start start 2024-03-25T18:21:32.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.435+0000 7f012d13e640 10 monclient: init 2024-03-25T18:21:32.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:32.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:32.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 10 monclient(hunting): picked mon.noname-b con 0x7f01280b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:32.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 10 monclient(hunting): picked mon.noname-a con 0x7f01280b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:32.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 10 monclient(hunting): picked mon.noname-c con 0x7f01280b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:32.438 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01280a96f0 con 0x7f01280b44b0 2024-03-25T18:21:32.438 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01280ab0e0 con 0x7f01280b1b00 2024-03-25T18:21:32.438 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0128145f70 con 0x7f01280b4e70 2024-03-25T18:21:32.438 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:32.438 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.436+0000 7f012d13e640 10 monclient(hunting): authenticate will time out at 7734.540928s 2024-03-25T18:21:32.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.441+0000 7f012659c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f01280b44b0 0x7f0128145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54276/0 (socket says 172.21.15.7:54276) 2024-03-25T18:21:32.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.441+0000 7f012659c640 1 -- 172.21.15.7:0/3540537714 learned_addr learned my addr 172.21.15.7:0/3540537714 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:32.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.456+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3393665013 0 0) 0x7f01280a96f0 con 0x7f01280b44b0 2024-03-25T18:21:32.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.456+0000 7f0125d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.456+0000 7f0125d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.456+0000 7f0125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.456+0000 7f0125d9b640 10 monclient(hunting): my global_id is 4521 2024-03-25T18:21:32.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.456+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0110003520 con 0x7f01280b44b0 2024-03-25T18:21:32.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.466+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3000625913 0 0) 0x7f01280ab0e0 con 0x7f01280b1b00 2024-03-25T18:21:32.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.466+0000 7f0125d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.466+0000 7f0125d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.466+0000 7f0125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.466+0000 7f0125d9b640 10 monclient(hunting): my global_id is 4522 2024-03-25T18:21:32.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.466+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01280a96f0 con 0x7f01280b1b00 2024-03-25T18:21:32.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.472+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504174315 0 0) 0x7f0128145f70 con 0x7f01280b4e70 2024-03-25T18:21:32.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.472+0000 7f0125d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.472+0000 7f0125d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.472+0000 7f0125d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.472+0000 7f0125d9b640 10 monclient(hunting): my global_id is 4553 2024-03-25T18:21:32.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.472+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f01280ab0e0 con 0x7f01280b4e70 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3298828133 0 0) 0x7f01280a96f0 con 0x7f01280b1b00 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 >> v1:172.21.15.7:6790/0 conn(0x7f01280b4e70 legacy=0x7f012813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 >> v1:172.21.15.7:6789/0 conn(0x7f01280b44b0 legacy=0x7f0128145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 1 monclient: found mon.noname-c 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01280a9b80 con 0x7f01280b1b00 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 10 monclient: _finish_auth 0 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f01200019f0 con 0x7f01280b1b00 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f0125d9b640 10 monclient: dump: 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f012d13e640 5 monclient: authenticate success, global_id 4522 2024-03-25T18:21:32.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.478+0000 7f012d13e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0120001b50 con 0x7f01280b1b00 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f0125d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f0125d9b640 1 -- 172.21.15.7:0/3540537714 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0120001f60 con 0x7f01280b1b00 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f0125d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f0125d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f0125d9b640 10 monclient: dump: 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.486 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.487 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:32.487 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:32.487 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:32.487 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:32.487 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:32.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 4 set_mon_vals no callback set 2024-03-25T18:21:32.488 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 10 monclient: shutdown 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 1 -- 172.21.15.7:0/3540537714 >> v1:172.21.15.183:6789/0 conn(0x7f01280b1b00 legacy=0x7f012814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 1 -- 172.21.15.7:0/3540537714 shutdown_connections 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 1 -- 172.21.15.7:0/3540537714 >> 172.21.15.7:0/3540537714 conn(0x7f01280617a0 msgr2=0x7f0128061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.485+0000 7f012d13e640 1 -- 172.21.15.7:0/3540537714 shutdown_connections 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.486+0000 7f012d13e640 1 -- 172.21.15.7:0/3540537714 wait complete. 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 10 monclient: build_initial_monmap 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 10 monclient: monmap: 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:32.489 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 1 librados: starting msgr at 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 1 librados: starting objecter 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 1 Processor -- start 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 1 -- start start 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 1 librados: setting wanted keys 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 1 librados: calling monclient init 2024-03-25T18:21:32.490 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.487+0000 7f012d13e640 10 monclient: init 2024-03-25T18:21:32.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:32.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:32.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 10 monclient(hunting): picked mon.noname-c con 0x7f01280b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:32.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 10 monclient(hunting): picked mon.noname-b con 0x7f01280b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:32.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 10 monclient(hunting): picked mon.noname-a con 0x7f01280b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:32.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01281363d0 con 0x7f01280b4e70 2024-03-25T18:21:32.491 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0128137640 con 0x7f01280b44b0 2024-03-25T18:21:32.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f01281378b0 con 0x7f01280b1b00 2024-03-25T18:21:32.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:32.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.488+0000 7f012d13e640 10 monclient(hunting): authenticate will time out at 7734.593192s 2024-03-25T18:21:32.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.493+0000 7f0126d9d640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f01280b1b00 0x7f01281357a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:60944/0 (socket says 172.21.15.7:60944) 2024-03-25T18:21:32.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.493+0000 7f0126d9d640 1 -- 172.21.15.7:0/1185758643 learned_addr learned my addr 172.21.15.7:0/1185758643 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:32.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.494+0000 7f010f7fe640 10 client.?.objecter ms_handle_connect 0x7f01280b44b0 2024-03-25T18:21:32.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.494+0000 7f010f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.497+0000 7f010f7fe640 10 client.?.objecter ms_handle_connect 0x7f01280b1b00 2024-03-25T18:21:32.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.497+0000 7f010f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.501+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378988306 0 0) 0x7f0128137640 con 0x7f01280b44b0 2024-03-25T18:21:32.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.501+0000 7f010f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.501+0000 7f010f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.501+0000 7f010f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.501+0000 7f010f7fe640 10 monclient(hunting): my global_id is 4525 2024-03-25T18:21:32.502 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.501+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0114004eb0 con 0x7f01280b44b0 2024-03-25T18:21:32.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.506+0000 7f010f7fe640 10 client.?.objecter ms_handle_connect 0x7f01280b4e70 2024-03-25T18:21:32.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.506+0000 7f010f7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:32.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.508+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2740145580 0 0) 0x7f01281378b0 con 0x7f01280b1b00 2024-03-25T18:21:32.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.508+0000 7f010f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.508+0000 7f010f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.508+0000 7f010f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.508+0000 7f010f7fe640 10 monclient(hunting): my global_id is 4556 2024-03-25T18:21:32.510 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.508+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0128137640 con 0x7f01280b1b00 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.511+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2866538385 0 0) 0x7f0114004eb0 con 0x7f01280b44b0 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.511+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01281378b0 con 0x7f01280b44b0 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.511+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f011c001950 con 0x7f01280b44b0 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.511+0000 7f010f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.511+0000 7f010f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.511+0000 7f010f7fe640 10 monclient(hunting): dump: 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:32.512 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:32.513 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:32.513 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.513 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.511+0000 7f010f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:32.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.515+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1809670678 0 0) 0x7f01281363d0 con 0x7f01280b4e70 2024-03-25T18:21:32.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.515+0000 7f010f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:21:32.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.515+0000 7f010f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:32.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.515+0000 7f010f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:32.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.515+0000 7f010f7fe640 10 monclient(hunting): my global_id is 4524 2024-03-25T18:21:32.516 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.515+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0114004eb0 con 0x7f01280b4e70 2024-03-25T18:21:32.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.525+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2433106148 0 0) 0x7f0128137640 con 0x7f01280b1b00 2024-03-25T18:21:32.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.525+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f01281363d0 con 0x7f01280b1b00 2024-03-25T18:21:32.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.525+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0118001950 con 0x7f01280b1b00 2024-03-25T18:21:32.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.525+0000 7f010f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.525+0000 7f010f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:32.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.525+0000 7f010f7fe640 10 monclient(hunting): dump: 2024-03-25T18:21:32.526 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:32.527 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.528 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.527+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3066492115 0 0) 0x7f0114004eb0 con 0x7f01280b4e70 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.527+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0128137640 con 0x7f01280b4e70 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.527+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0120001e60 con 0x7f01280b4e70 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.527+0000 7f010f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.527+0000 7f010f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.527+0000 7f010f7fe640 10 monclient(hunting): dump: 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:32.529 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1554279517 0 0) 0x7f01281378b0 con 0x7f01280b44b0 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 >> v1:172.21.15.7:6790/0 conn(0x7f01280b1b00 legacy=0x7f01281357a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 >> v1:172.21.15.7:6789/0 conn(0x7f01280b4e70 legacy=0x7f01281e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 1 monclient: found mon.b 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0128137b90 con 0x7f01280b44b0 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f010f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 5 monclient: authenticate success, global_id 4525 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 10 monclient: _renew_subs 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:32.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0128136860 con 0x7f01280b44b0 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 10 monclient: _renew_subs 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f01281386c0 con 0x7f01280b44b0 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.529+0000 7f012d13e640 1 librados: init done 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.530+0000 7f012d13e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.530+0000 7f012d13e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.530+0000 7f012d13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:32.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.530+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0128136580 con 0x7f01280b44b0 2024-03-25T18:21:32.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.799+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f011c001d60 con 0x7f01280b44b0 2024-03-25T18:21:32.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.799+0000 7f010f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:32.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.799+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f011c0047e0 con 0x7f01280b44b0 2024-03-25T18:21:32.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.799+0000 7f010f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:32.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.799+0000 7f010f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.799+0000 7f010f7fe640 10 monclient: dump: 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:32.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.799+0000 7f012559a640 4 set_mon_vals no callback set 2024-03-25T18:21:32.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.800+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f011c01a830 con 0x7f01280b44b0 2024-03-25T18:21:32.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.801+0000 7f0126d9d640 10 monclient: get_auth_request con 0x7f011406e110 auth_method 0 2024-03-25T18:21:32.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.801+0000 7f010f7fe640 10 client.4525.objecter ms_handle_connect 0x7f011406e110 2024-03-25T18:21:32.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f011c0545b0 con 0x7f01280b44b0 2024-03-25T18:21:32.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 10 client.4525.objecter ms_dispatch 0x7f0128000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:32.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:32.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:32.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-25T18:21:32.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f011c04fef0 con 0x7f01280b44b0 2024-03-25T18:21:32.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:32.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:32.804+0000 7f010f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:33.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.025+0000 7f012d13e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.025+0000 7f012d13e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.025+0000 7f012d13e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:33.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.025+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0128136140 con 0x7f01280b44b0 2024-03-25T18:21:33.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.105+0000 7f010f7fe640 1 -- 172.21.15.7:0/1185758643 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f011c051ff0 con 0x7f01280b44b0 2024-03-25T18:21:33.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.105+0000 7f010f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:33.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.105+0000 7f010f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:33.107 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:21:33.107 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 10 librados: watch_flush enter 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 10 librados: watch_flush exit 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 10 client.4525.objecter successfully canceled tick 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 >> v1:172.21.15.7:6800/1717564767 conn(0x7f011406e110 legacy=0x7f01140705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 10 monclient: shutdown 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:33.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 >> v1:172.21.15.183:6789/0 conn(0x7f01280b44b0 legacy=0x7f0128137fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:33.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 shutdown_connections 2024-03-25T18:21:33.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 >> 172.21.15.7:0/1185758643 conn(0x7f01280617a0 msgr2=0x7f01280b6580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:33.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.107+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 shutdown_connections 2024-03-25T18:21:33.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.108+0000 7f012d13e640 1 -- 172.21.15.7:0/1185758643 wait complete. 2024-03-25T18:21:33.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:33.108+0000 7f012d13e640 1 librados: shutdown 2024-03-25T18:21:33.120 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:39.927 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:40.041 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.040+0000 7f712b5cf640 10 monclient: get_monmap_and_config 2024-03-25T18:21:40.041 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.040+0000 7f712b5cf640 10 monclient: build_initial_monmap 2024-03-25T18:21:40.041 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.040+0000 7f712b5cf640 10 monclient: monmap: 2024-03-25T18:21:40.041 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:40.041 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:21:40.041406+0000 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:21:40.041406+0000 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:21:40.042 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:40.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.041+0000 7f712b5cf640 1 Processor -- start 2024-03-25T18:21:40.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.041+0000 7f712b5cf640 1 -- start start 2024-03-25T18:21:40.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.041+0000 7f712b5cf640 10 monclient: init 2024-03-25T18:21:40.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:40.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:40.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 10 monclient(hunting): picked mon.noname-b con 0x7f712406ae10 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:40.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 10 monclient(hunting): picked mon.noname-c con 0x7f712413da60 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:40.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f7124143d40 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:40.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240afdc0 con 0x7f7124143d40 2024-03-25T18:21:40.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240b1160 con 0x7f712413da60 2024-03-25T18:21:40.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240b1440 con 0x7f712406ae10 2024-03-25T18:21:40.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:40.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.042+0000 7f712b5cf640 10 monclient(hunting): authenticate will time out at 7742.146959s 2024-03-25T18:21:40.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.055+0000 7f7129b6c640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f7124143d40 0x7f712414b9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:58946/0 (socket says 172.21.15.7:58946) 2024-03-25T18:21:40.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.055+0000 7f7129b6c640 1 -- 172.21.15.7:0/1554517943 learned_addr learned my addr 172.21.15.7:0/1554517943 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:40.066 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.065+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3122361479 0 0) 0x7f71240afdc0 con 0x7f7124143d40 2024-03-25T18:21:40.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.065+0000 7f711bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.065+0000 7f711bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.065+0000 7f711bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.065+0000 7f711bfff640 10 monclient(hunting): my global_id is 4527 2024-03-25T18:21:40.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.066+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7110003520 con 0x7f7124143d40 2024-03-25T18:21:40.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.067+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4011139067 0 0) 0x7f71240b1440 con 0x7f712406ae10 2024-03-25T18:21:40.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.067+0000 7f711bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.067+0000 7f711bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.067+0000 7f711bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.067+0000 7f711bfff640 10 monclient(hunting): my global_id is 4559 2024-03-25T18:21:40.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.067+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71240afdc0 con 0x7f712406ae10 2024-03-25T18:21:40.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.071+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3820457621 0 0) 0x7f71240b1160 con 0x7f712413da60 2024-03-25T18:21:40.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.071+0000 7f711bfff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.071+0000 7f711bfff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.071+0000 7f711bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.071+0000 7f711bfff640 10 monclient(hunting): my global_id is 4528 2024-03-25T18:21:40.072 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.071+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71240b1440 con 0x7f712413da60 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1549783343 0 0) 0x7f71240b1440 con 0x7f712413da60 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 >> v1:172.21.15.7:6790/0 conn(0x7f712406ae10 legacy=0x7f712413d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 >> v1:172.21.15.7:6789/0 conn(0x7f7124143d40 legacy=0x7f712414b9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 1 monclient: found mon.noname-c 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71240b0250 con 0x7f712413da60 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 10 monclient: _finish_auth 0 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:40.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f71140015b0 con 0x7f712413da60 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f711bfff640 10 monclient: dump: 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f712b5cf640 5 monclient: authenticate success, global_id 4528 2024-03-25T18:21:40.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.082+0000 7f712b5cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7114001710 con 0x7f712413da60 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f711bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f711bfff640 1 -- 172.21.15.7:0/1554517943 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7114001b60 con 0x7f712413da60 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f712b5cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f712b5cf640 4 set_mon_vals no callback set 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f711bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f711bfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f711bfff640 10 monclient: dump: 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.090 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.091 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.091 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:40.091 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:40.091 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:40.091 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:40.091 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:40.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f712b5cf640 10 monclient: shutdown 2024-03-25T18:21:40.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f712b5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:40.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.089+0000 7f712b5cf640 1 -- 172.21.15.7:0/1554517943 >> v1:172.21.15.183:6789/0 conn(0x7f712413da60 legacy=0x7f7124143630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.090+0000 7f712b5cf640 1 -- 172.21.15.7:0/1554517943 shutdown_connections 2024-03-25T18:21:40.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.090+0000 7f712b5cf640 1 -- 172.21.15.7:0/1554517943 >> 172.21.15.7:0/1554517943 conn(0x7f71240b4bd0 msgr2=0x7f71240b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:40.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.090+0000 7f712b5cf640 1 -- 172.21.15.7:0/1554517943 shutdown_connections 2024-03-25T18:21:40.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.090+0000 7f712b5cf640 1 -- 172.21.15.7:0/1554517943 wait complete. 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.090+0000 7f712b5cf640 10 monclient: build_initial_monmap 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.090+0000 7f712b5cf640 10 monclient: monmap: 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.091+0000 7f712b5cf640 1 librados: starting msgr at 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.091+0000 7f712b5cf640 1 librados: starting objecter 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.091+0000 7f712b5cf640 1 Processor -- start 2024-03-25T18:21:40.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.091+0000 7f712b5cf640 1 -- start start 2024-03-25T18:21:40.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.091+0000 7f712b5cf640 1 librados: setting wanted keys 2024-03-25T18:21:40.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.091+0000 7f712b5cf640 1 librados: calling monclient init 2024-03-25T18:21:40.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.091+0000 7f712b5cf640 10 monclient: init 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 10 monclient(hunting): picked mon.noname-a con 0x7f712406ae10 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 10 monclient(hunting): picked mon.noname-c con 0x7f712413da60 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 10 monclient(hunting): picked mon.noname-b con 0x7f7124143d40 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7124140fd0 con 0x7f712406ae10 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240ae5b0 con 0x7f7124143d40 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71240ae740 con 0x7f712413da60 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:40.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.092+0000 7f712b5cf640 10 monclient(hunting): authenticate will time out at 7742.196917s 2024-03-25T18:21:40.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.097+0000 7f7129b6c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7124143d40 0x7f71241d52f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:56254/0 (socket says 172.21.15.7:56254) 2024-03-25T18:21:40.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.097+0000 7f7129b6c640 1 -- 172.21.15.7:0/3532274234 learned_addr learned my addr 172.21.15.7:0/3532274234 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:40.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.098+0000 7f7119ffb640 10 client.?.objecter ms_handle_connect 0x7f7124143d40 2024-03-25T18:21:40.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.098+0000 7f7119ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 10 client.?.objecter ms_handle_connect 0x7f712406ae10 2024-03-25T18:21:40.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2180112377 0 0) 0x7f71240ae5b0 con 0x7f7124143d40 2024-03-25T18:21:40.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 10 monclient(hunting): my global_id is 4531 2024-03-25T18:21:40.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.103+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7108004eb0 con 0x7f7124143d40 2024-03-25T18:21:40.114 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.113+0000 7f7119ffb640 10 client.?.objecter ms_handle_connect 0x7f712413da60 2024-03-25T18:21:40.114 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.113+0000 7f7119ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:40.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.114+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1177798176 0 0) 0x7f7124140fd0 con 0x7f712406ae10 2024-03-25T18:21:40.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.114+0000 7f7119ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.114+0000 7f7119ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.114+0000 7f7119ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.114+0000 7f7119ffb640 10 monclient(hunting): my global_id is 4530 2024-03-25T18:21:40.115 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.114+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71240ae5b0 con 0x7f712406ae10 2024-03-25T18:21:40.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.123+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 962494509 0 0) 0x7f7108004eb0 con 0x7f7124143d40 2024-03-25T18:21:40.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.123+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7124140fd0 con 0x7f7124143d40 2024-03-25T18:21:40.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.123+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f711c001b70 con 0x7f7124143d40 2024-03-25T18:21:40.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.123+0000 7f7119ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.123+0000 7f7119ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:40.124 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.123+0000 7f7119ffb640 10 monclient(hunting): dump: 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:40.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.123+0000 7f7119ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:40.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.126+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1083303623 0 0) 0x7f71240ae740 con 0x7f712413da60 2024-03-25T18:21:40.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.126+0000 7f7119ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:40.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.126+0000 7f7119ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:40.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.126+0000 7f7119ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:40.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.126+0000 7f7119ffb640 10 monclient(hunting): my global_id is 4562 2024-03-25T18:21:40.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.126+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7108004eb0 con 0x7f712413da60 2024-03-25T18:21:40.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2801285145 0 0) 0x7f7124140fd0 con 0x7f7124143d40 2024-03-25T18:21:40.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 >> v1:172.21.15.7:6790/0 conn(0x7f712413da60 legacy=0x7f7124137fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 >> v1:172.21.15.7:6789/0 conn(0x7f712406ae10 legacy=0x7f71241406c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 1 monclient: found mon.b 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7124140050 con 0x7f7124143d40 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f7119ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 5 monclient: authenticate success, global_id 4531 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 10 monclient: _renew_subs 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7124140300 con 0x7f7124143d40 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 10 client.4531.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 10 monclient: _renew_subs 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f71241388d0 con 0x7f7124143d40 2024-03-25T18:21:40.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.130+0000 7f712b5cf640 1 librados: init done 2024-03-25T18:21:40.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.131+0000 7f712b5cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:40.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.131+0000 7f712b5cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:40.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.131+0000 7f712b5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:40.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.131+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7124140dd0 con 0x7f7124143d40 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.406+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f711c002060 con 0x7f7124143d40 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.406+0000 7f7119ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.406+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f711c004d00 con 0x7f7124143d40 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.406+0000 7f7119ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.407+0000 7f7119ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.407+0000 7f7119ffb640 10 monclient: dump: 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:40.408 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:40.409 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:40.409 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:40.409 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:40.409 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:40.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.407+0000 7f711b7fe640 4 set_mon_vals no callback set 2024-03-25T18:21:40.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.408+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f711c01acd0 con 0x7f7124143d40 2024-03-25T18:21:40.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.408+0000 7f712936b640 10 monclient: get_auth_request con 0x7f710806ddb0 auth_method 0 2024-03-25T18:21:40.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.409+0000 7f7119ffb640 10 client.4531.objecter ms_handle_connect 0x7f710806ddb0 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f711c054ab0 con 0x7f7124143d40 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 10 client.4531.objecter ms_dispatch 0x7f7124000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 3 client.4531.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 3 client.4531.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 20 client.4531.objecter dump_active .. 0 homeless 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f711c0503f0 con 0x7f7124143d40 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:40.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.412+0000 7f7119ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:40.660 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.659+0000 7f712b5cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.660 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.659+0000 7f712b5cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.659+0000 7f712b5cf640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:40.661 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.659+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f712406b1f0 con 0x7f7124143d40 2024-03-25T18:21:40.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.781+0000 7f7119ffb640 1 -- 172.21.15.7:0/3532274234 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f711c0524f0 con 0x7f7124143d40 2024-03-25T18:21:40.783 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.781+0000 7f7119ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:40.783 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.781+0000 7f7119ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:40.783 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:21:40.783 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:40.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 10 librados: watch_flush enter 2024-03-25T18:21:40.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 10 librados: watch_flush exit 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 20 client.4531.objecter shutdown clearing up homeless session... 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 10 client.4531.objecter successfully canceled tick 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 >> v1:172.21.15.7:6800/1717564767 conn(0x7f710806ddb0 legacy=0x7f7108070270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 10 monclient: shutdown 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 >> v1:172.21.15.183:6789/0 conn(0x7f7124143d40 legacy=0x7f71241d52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 shutdown_connections 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 >> 172.21.15.7:0/3532274234 conn(0x7f71240b4bd0 msgr2=0x7f71240b4070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 shutdown_connections 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 1 -- 172.21.15.7:0/3532274234 wait complete. 2024-03-25T18:21:40.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:40.784+0000 7f712b5cf640 1 librados: shutdown 2024-03-25T18:21:40.796 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:47.602 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:47.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.707+0000 7f946e93e640 10 monclient: get_monmap_and_config 2024-03-25T18:21:47.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.707+0000 7f946e93e640 10 monclient: build_initial_monmap 2024-03-25T18:21:47.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.707+0000 7f946e93e640 10 monclient: monmap: 2024-03-25T18:21:47.708 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:47.708 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:47.708 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:21:47.708481+0000 2024-03-25T18:21:47.708 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:21:47.708481+0000 2024-03-25T18:21:47.709 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:47.709 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:47.709 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:47.709 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:21:47.709 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:21:47.709 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 1 Processor -- start 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 1 -- start start 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient: init 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient(hunting): picked mon.noname-b con 0x7f94680a98d0 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient(hunting): picked mon.noname-a con 0x7f94680b2a60 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient(hunting): picked mon.noname-c con 0x7f946814bc00 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:47.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94680b2e80 con 0x7f94680b2a60 2024-03-25T18:21:47.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94681431c0 con 0x7f946814bc00 2024-03-25T18:21:47.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f94681422f0 con 0x7f94680a98d0 2024-03-25T18:21:47.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:47.711 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.708+0000 7f946e93e640 10 monclient(hunting): authenticate will time out at 7749.813476s 2024-03-25T18:21:47.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.718+0000 7f946cedb640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f946814bc00 0x7f946814dff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:35798/0 (socket says 172.21.15.7:35798) 2024-03-25T18:21:47.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.718+0000 7f946cedb640 1 -- 172.21.15.7:0/420136644 learned_addr learned my addr 172.21.15.7:0/420136644 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:47.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.737+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3720622367 0 0) 0x7f94681422f0 con 0x7f94680a98d0 2024-03-25T18:21:47.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.737+0000 7f9466ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.737+0000 7f9466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.737+0000 7f9466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.737+0000 7f9466ffd640 10 monclient(hunting): my global_id is 4565 2024-03-25T18:21:47.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.737+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9450003520 con 0x7f94680a98d0 2024-03-25T18:21:47.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.739+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4035882464 0 0) 0x7f94681431c0 con 0x7f946814bc00 2024-03-25T18:21:47.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.739+0000 7f9466ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.739+0000 7f9466ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.741 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.739+0000 7f9466ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.741 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.739+0000 7f9466ffd640 10 monclient(hunting): my global_id is 4534 2024-03-25T18:21:47.741 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.739+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f94681422f0 con 0x7f946814bc00 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.756+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2629964 0 0) 0x7f94681422f0 con 0x7f946814bc00 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 >> v1:172.21.15.7:6790/0 conn(0x7f94680a98d0 legacy=0x7f9468142ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 >> v1:172.21.15.7:6789/0 conn(0x7f94680b2a60 legacy=0x7f946813f440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 1 monclient: found mon.noname-c 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9468142590 con 0x7f946814bc00 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 10 monclient: _finish_auth 0 2024-03-25T18:21:47.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f94600019f0 con 0x7f946814bc00 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f9466ffd640 10 monclient: dump: 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:47.759 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f946e93e640 5 monclient: authenticate success, global_id 4534 2024-03-25T18:21:47.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.757+0000 7f946e93e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:47.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9460001b50 con 0x7f946814bc00 2024-03-25T18:21:47.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f9466ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:47.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f9466ffd640 1 -- 172.21.15.7:0/420136644 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9460001f20 con 0x7f946814bc00 2024-03-25T18:21:47.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f9466ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f9466ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:47.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f9466ffd640 10 monclient: dump: 2024-03-25T18:21:47.767 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f946e93e640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:47.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.766+0000 7f946e93e640 4 set_mon_vals no callback set 2024-03-25T18:21:47.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.767+0000 7f946e93e640 10 monclient: shutdown 2024-03-25T18:21:47.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.767+0000 7f946e93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:47.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.767+0000 7f946e93e640 1 -- 172.21.15.7:0/420136644 >> v1:172.21.15.183:6789/0 conn(0x7f946814bc00 legacy=0x7f946814dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.767+0000 7f946e93e640 1 -- 172.21.15.7:0/420136644 shutdown_connections 2024-03-25T18:21:47.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.767+0000 7f946e93e640 1 -- 172.21.15.7:0/420136644 >> 172.21.15.7:0/420136644 conn(0x7f94680b4b60 msgr2=0x7f94680b4f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:47.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.767+0000 7f946e93e640 1 -- 172.21.15.7:0/420136644 shutdown_connections 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.767+0000 7f946e93e640 1 -- 172.21.15.7:0/420136644 wait complete. 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.768+0000 7f946e93e640 10 monclient: build_initial_monmap 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.768+0000 7f946e93e640 10 monclient: monmap: 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:21:47.770 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:47.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.768+0000 7f946e93e640 1 librados: starting msgr at 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.768+0000 7f946e93e640 1 librados: starting objecter 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 1 Processor -- start 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 1 -- start start 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 1 librados: setting wanted keys 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 1 librados: calling monclient init 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 10 monclient: init 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.769+0000 7f946e93e640 10 monclient(hunting): picked mon.noname-a con 0x7f94680a98d0 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.770+0000 7f946e93e640 10 monclient(hunting): picked mon.noname-b con 0x7f94680b2a60 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.770+0000 7f946e93e640 10 monclient(hunting): picked mon.noname-c con 0x7f946814bc00 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.770+0000 7f946e93e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946813bd80 con 0x7f94680a98d0 2024-03-25T18:21:47.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.770+0000 7f946e93e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9468148610 con 0x7f94680b2a60 2024-03-25T18:21:47.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.770+0000 7f946e93e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f946813b910 con 0x7f946814bc00 2024-03-25T18:21:47.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.770+0000 7f946e93e640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:47.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.770+0000 7f946e93e640 10 monclient(hunting): authenticate will time out at 7749.874795s 2024-03-25T18:21:47.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.772+0000 7f94677fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f94680b2a60 0x7f94681db4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:35814/0 (socket says 172.21.15.7:35814) 2024-03-25T18:21:47.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.772+0000 7f94677fe640 1 -- 172.21.15.7:0/1073644451 learned_addr learned my addr 172.21.15.7:0/1073644451 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:47.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.773+0000 7f9464ff9640 10 client.?.objecter ms_handle_connect 0x7f94680b2a60 2024-03-25T18:21:47.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.773+0000 7f9464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:47.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.778+0000 7f9464ff9640 10 client.?.objecter ms_handle_connect 0x7f946814bc00 2024-03-25T18:21:47.780 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.778+0000 7f9464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:47.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.781+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431630566 0 0) 0x7f9468148610 con 0x7f94680b2a60 2024-03-25T18:21:47.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.781+0000 7f9464ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.781+0000 7f9464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.782 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.781+0000 7f9464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.783 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.781+0000 7f9464ff9640 10 monclient(hunting): my global_id is 4537 2024-03-25T18:21:47.783 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.781+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454004eb0 con 0x7f94680b2a60 2024-03-25T18:21:47.789 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.788+0000 7f9464ff9640 10 client.?.objecter ms_handle_connect 0x7f94680a98d0 2024-03-25T18:21:47.789 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.788+0000 7f9464ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:47.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.794+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2930286114 0 0) 0x7f9454004eb0 con 0x7f94680b2a60 2024-03-25T18:21:47.795 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.794+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9468148610 con 0x7f94680b2a60 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.794+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f945c001a30 con 0x7f94680b2a60 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.794+0000 7f9464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.794+0000 7f9464ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.794+0000 7f9464ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:47.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.794+0000 7f9464ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:47.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.796+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3889094066 0 0) 0x7f946813b910 con 0x7f946814bc00 2024-03-25T18:21:47.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.796+0000 7f9464ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.796+0000 7f9464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.796+0000 7f9464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.796+0000 7f9464ff9640 10 monclient(hunting): my global_id is 4568 2024-03-25T18:21:47.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.796+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454004eb0 con 0x7f946814bc00 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.808+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3388852004 0 0) 0x7f9454004eb0 con 0x7f946814bc00 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.808+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f946813b910 con 0x7f946814bc00 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.808+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f9460001e60 con 0x7f946814bc00 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.808+0000 7f9464ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.808+0000 7f9464ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.808+0000 7f9464ff9640 10 monclient(hunting): dump: 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:47.809 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:47.810 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:47.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.811+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073305779 0 0) 0x7f946813bd80 con 0x7f94680a98d0 2024-03-25T18:21:47.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.811+0000 7f9464ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:21:47.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.811+0000 7f9464ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:47.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.811+0000 7f9464ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:47.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.811+0000 7f9464ff9640 10 monclient(hunting): my global_id is 4536 2024-03-25T18:21:47.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.811+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9454004eb0 con 0x7f94680a98d0 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 35712756 0 0) 0x7f9468148610 con 0x7f94680b2a60 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 >> v1:172.21.15.7:6790/0 conn(0x7f946814bc00 legacy=0x7f9468138110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 >> v1:172.21.15.7:6789/0 conn(0x7f94680a98d0 legacy=0x7f946813a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 1 monclient: found mon.b 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f946813baa0 con 0x7f94680b2a60 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 10 monclient: _finish_auth 0 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f9464ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 5 monclient: authenticate success, global_id 4537 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 10 monclient: _renew_subs 2024-03-25T18:21:47.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:47.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9468146a50 con 0x7f94680b2a60 2024-03-25T18:21:47.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 10 client.4537.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:47.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 10 monclient: _renew_subs 2024-03-25T18:21:47.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:47.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f94681dbbe0 con 0x7f94680b2a60 2024-03-25T18:21:47.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.815+0000 7f946e93e640 1 librados: init done 2024-03-25T18:21:47.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.816+0000 7f946e93e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:47.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.816+0000 7f946e93e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:47.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.816+0000 7f946e93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:47.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:47.816+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94680b1d00 con 0x7f94680b2a60 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.096+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f945c001cb0 con 0x7f94680b2a60 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.096+0000 7f9464ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.096+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f945c003a60 con 0x7f94680b2a60 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.096+0000 7f9464ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:48.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.096+0000 7f9464ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.096+0000 7f9464ff9640 10 monclient: dump: 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:48.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.096+0000 7f94667fc640 4 set_mon_vals no callback set 2024-03-25T18:21:48.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.097+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f945c003c40 con 0x7f94680b2a60 2024-03-25T18:21:48.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.097+0000 7f9467fff640 10 monclient: get_auth_request con 0x7f945406dfb0 auth_method 0 2024-03-25T18:21:48.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.097+0000 7f9464ff9640 10 client.4537.objecter ms_handle_connect 0x7f945406dfb0 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.105+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f945c0548a0 con 0x7f94680b2a60 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.105+0000 7f9464ff9640 10 client.4537.objecter ms_dispatch 0x7f9468000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.105+0000 7f9464ff9640 3 client.4537.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.105+0000 7f9464ff9640 3 client.4537.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.105+0000 7f9464ff9640 20 client.4537.objecter dump_active .. 0 homeless 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.106+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f945c0503b0 con 0x7f94680b2a60 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.106+0000 7f9464ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:48.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.106+0000 7f9464ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:48.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.364+0000 7f946e93e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:48.366 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.364+0000 7f946e93e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:48.366 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.364+0000 7f946e93e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:21:48.366 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.364+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94680ab020 con 0x7f94680b2a60 2024-03-25T18:21:48.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.405+0000 7f9464ff9640 1 -- 172.21.15.7:0/1073644451 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f945c0522e0 con 0x7f94680b2a60 2024-03-25T18:21:48.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.405+0000 7f9464ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:48.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.405+0000 7f9464ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:48.407 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:21:48.407 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:48.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.407+0000 7f946e93e640 10 librados: watch_flush enter 2024-03-25T18:21:48.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.407+0000 7f946e93e640 10 librados: watch_flush exit 2024-03-25T18:21:48.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.407+0000 7f946e93e640 20 client.4537.objecter shutdown clearing up homeless session... 2024-03-25T18:21:48.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.407+0000 7f946e93e640 10 client.4537.objecter successfully canceled tick 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 >> v1:172.21.15.7:6800/1717564767 conn(0x7f945406dfb0 legacy=0x7f9454070470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 10 monclient: shutdown 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 >> v1:172.21.15.183:6789/0 conn(0x7f94680b2a60 legacy=0x7f94681db4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 shutdown_connections 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 >> 172.21.15.7:0/1073644451 conn(0x7f94680b4b60 msgr2=0x7f94680b78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 shutdown_connections 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 1 -- 172.21.15.7:0/1073644451 wait complete. 2024-03-25T18:21:48.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:48.408+0000 7f946e93e640 1 librados: shutdown 2024-03-25T18:21:48.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:21:48.996 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:55.633 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.045 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:55.682 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.100 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:07:55.737 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.109 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:55.735 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.124 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:55.750 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.125 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:55.762 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.129 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:55.766 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.155 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:07:55.782 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:49.169 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:07:55.806 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:21:55.228 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.344+0000 7fdf6f895640 10 monclient: get_monmap_and_config 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.344+0000 7fdf6f895640 10 monclient: build_initial_monmap 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.344+0000 7fdf6f895640 10 monclient: monmap: 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:21:55.345169+0000 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:21:55.345169+0000 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:55.345 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:55.346 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:55.346 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:21:55.346 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:21:55.346 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.344+0000 7fdf6f895640 1 Processor -- start 2024-03-25T18:21:55.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 1 -- start start 2024-03-25T18:21:55.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient: init 2024-03-25T18:21:55.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:55.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:21:55.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient(hunting): picked mon.noname-c con 0x7fdf680b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:55.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient(hunting): picked mon.noname-a con 0x7fdf680b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:55.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient(hunting): picked mon.noname-b con 0x7fdf680b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:55.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680a96f0 con 0x7fdf680b44b0 2024-03-25T18:21:55.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf680ab0e0 con 0x7fdf680b4e70 2024-03-25T18:21:55.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf68145f70 con 0x7fdf680b1b00 2024-03-25T18:21:55.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:55.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.345+0000 7fdf6f895640 10 monclient(hunting): authenticate will time out at 7757.450546s 2024-03-25T18:21:55.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.349+0000 7fdf6de32640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fdf680b1b00 0x7fdf6814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:39210/0 (socket says 172.21.15.7:39210) 2024-03-25T18:21:55.350 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.349+0000 7fdf6de32640 1 -- 172.21.15.7:0/1476938134 learned_addr learned my addr 172.21.15.7:0/1476938134 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:55.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.371+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 172636481 0 0) 0x7fdf68145f70 con 0x7fdf680b1b00 2024-03-25T18:21:55.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.371+0000 7fdf5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.371+0000 7fdf5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.371+0000 7fdf5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.371+0000 7fdf5ffff640 10 monclient(hunting): my global_id is 4571 2024-03-25T18:21:55.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.371+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf54003520 con 0x7fdf680b1b00 2024-03-25T18:21:55.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.380+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 846580062 0 0) 0x7fdf680a96f0 con 0x7fdf680b44b0 2024-03-25T18:21:55.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.380+0000 7fdf5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.380+0000 7fdf5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.380+0000 7fdf5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.380+0000 7fdf5ffff640 10 monclient(hunting): my global_id is 4539 2024-03-25T18:21:55.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.380+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf68145f70 con 0x7fdf680b44b0 2024-03-25T18:21:55.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.382+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 817686671 0 0) 0x7fdf680ab0e0 con 0x7fdf680b4e70 2024-03-25T18:21:55.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.383+0000 7fdf5ffff640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.383+0000 7fdf5ffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.383+0000 7fdf5ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.383+0000 7fdf5ffff640 10 monclient(hunting): my global_id is 4540 2024-03-25T18:21:55.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.383+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf680a96f0 con 0x7fdf680b4e70 2024-03-25T18:21:55.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3424918654 0 0) 0x7fdf54003520 con 0x7fdf680b1b00 2024-03-25T18:21:55.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 >> v1:172.21.15.183:6789/0 conn(0x7fdf680b4e70 legacy=0x7fdf6813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 >> v1:172.21.15.7:6789/0 conn(0x7fdf680b44b0 legacy=0x7fdf68145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 10 monclient: _finish_hunting 0 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 1 monclient: found mon.noname-b 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf680a9b80 con 0x7fdf680b1b00 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 10 monclient: _finish_auth 0 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 10 monclient: _check_auth_tickets 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdf60001a10 con 0x7fdf680b1b00 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf5ffff640 10 monclient: dump: 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:55.386 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf6f895640 5 monclient: authenticate success, global_id 4571 2024-03-25T18:21:55.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.384+0000 7fdf6f895640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf60001b70 con 0x7fdf680b1b00 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf5ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf5ffff640 1 -- 172.21.15.7:0/1476938134 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdf60001f80 con 0x7fdf680b1b00 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf6f895640 10 monclient: get_monmap_and_config success 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf6f895640 4 set_mon_vals no callback set 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf5ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf5ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf5ffff640 10 monclient: dump: 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:55.403 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:55.404 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf6f895640 10 monclient: shutdown 2024-03-25T18:21:55.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf6f895640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:55.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.402+0000 7fdf6f895640 1 -- 172.21.15.7:0/1476938134 >> v1:172.21.15.7:6790/0 conn(0x7fdf680b1b00 legacy=0x7fdf6814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.403+0000 7fdf6f895640 1 -- 172.21.15.7:0/1476938134 shutdown_connections 2024-03-25T18:21:55.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.403+0000 7fdf6f895640 1 -- 172.21.15.7:0/1476938134 >> 172.21.15.7:0/1476938134 conn(0x7fdf680617a0 msgr2=0x7fdf68061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.403+0000 7fdf6f895640 1 -- 172.21.15.7:0/1476938134 shutdown_connections 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.403+0000 7fdf6f895640 1 -- 172.21.15.7:0/1476938134 wait complete. 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.403+0000 7fdf6f895640 10 monclient: build_initial_monmap 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.403+0000 7fdf6f895640 10 monclient: monmap: 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 1 librados: starting msgr at 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 1 librados: starting objecter 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 1 Processor -- start 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 1 -- start start 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 1 librados: setting wanted keys 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 1 librados: calling monclient init 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 10 monclient: init 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 10 monclient: _reopen_session rank -1 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.404+0000 7fdf6f895640 10 monclient(hunting): picked mon.noname-c con 0x7fdf680b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.405+0000 7fdf6f895640 10 monclient(hunting): picked mon.noname-b con 0x7fdf680b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.405+0000 7fdf6f895640 10 monclient(hunting): picked mon.noname-a con 0x7fdf680b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.405+0000 7fdf6f895640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf681452b0 con 0x7fdf680b4e70 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.405+0000 7fdf6f895640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6813c3b0 con 0x7fdf680b44b0 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.405+0000 7fdf6f895640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf6813c690 con 0x7fdf680b1b00 2024-03-25T18:21:55.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.405+0000 7fdf6f895640 10 monclient(hunting): _renew_subs 2024-03-25T18:21:55.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.405+0000 7fdf6f895640 10 monclient(hunting): authenticate will time out at 7757.509922s 2024-03-25T18:21:55.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.409+0000 7fdf6ce30640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fdf680b44b0 0x7fdf68143b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:52826/0 (socket says 172.21.15.7:52826) 2024-03-25T18:21:55.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.409+0000 7fdf6ce30640 1 -- 172.21.15.7:0/1820198213 learned_addr learned my addr 172.21.15.7:0/1820198213 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:21:55.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.410+0000 7fdf5dffb640 10 client.?.objecter ms_handle_connect 0x7fdf680b4e70 2024-03-25T18:21:55.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.410+0000 7fdf5dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:55.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.411+0000 7fdf5dffb640 10 client.?.objecter ms_handle_connect 0x7fdf680b44b0 2024-03-25T18:21:55.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.411+0000 7fdf5dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:55.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.420+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2194446622 0 0) 0x7fdf6813c3b0 con 0x7fdf680b44b0 2024-03-25T18:21:55.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.420+0000 7fdf5dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.420+0000 7fdf5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.420+0000 7fdf5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.420+0000 7fdf5dffb640 10 monclient(hunting): my global_id is 4543 2024-03-25T18:21:55.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.420+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c004eb0 con 0x7fdf680b44b0 2024-03-25T18:21:55.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.421+0000 7fdf5dffb640 10 client.?.objecter ms_handle_connect 0x7fdf680b1b00 2024-03-25T18:21:55.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.421+0000 7fdf5dffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:21:55.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.429+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342187012 0 0) 0x7fdf681452b0 con 0x7fdf680b4e70 2024-03-25T18:21:55.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.429+0000 7fdf5dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.429+0000 7fdf5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.429+0000 7fdf5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.429+0000 7fdf5dffb640 10 monclient(hunting): my global_id is 4542 2024-03-25T18:21:55.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.429+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf6813c3b0 con 0x7fdf680b4e70 2024-03-25T18:21:55.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.431+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2515917530 0 0) 0x7fdf4c004eb0 con 0x7fdf680b44b0 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.431+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf681452b0 con 0x7fdf680b44b0 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.431+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdf58001a10 con 0x7fdf680b44b0 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.431+0000 7fdf5dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.431+0000 7fdf5dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.431+0000 7fdf5dffb640 10 monclient(hunting): dump: 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:55.433 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.431+0000 7fdf5dffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:21:55.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.433+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 540282644 0 0) 0x7fdf6813c690 con 0x7fdf680b1b00 2024-03-25T18:21:55.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.433+0000 7fdf5dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:21:55.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.433+0000 7fdf5dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:21:55.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.433+0000 7fdf5dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:21:55.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.433+0000 7fdf5dffb640 10 monclient(hunting): my global_id is 4574 2024-03-25T18:21:55.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.433+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf4c004eb0 con 0x7fdf680b1b00 2024-03-25T18:21:55.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.435+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3007899102 0 0) 0x7fdf6813c3b0 con 0x7fdf680b4e70 2024-03-25T18:21:55.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.435+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf6813c690 con 0x7fdf680b4e70 2024-03-25T18:21:55.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.435+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdf60001e80 con 0x7fdf680b4e70 2024-03-25T18:21:55.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.435+0000 7fdf5dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.435+0000 7fdf5dffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.435+0000 7fdf5dffb640 10 monclient(hunting): dump: 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:55.437 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.440+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2513925419 0 0) 0x7fdf4c004eb0 con 0x7fdf680b1b00 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.440+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdf6813c3b0 con 0x7fdf680b1b00 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.440+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdf64001790 con 0x7fdf680b1b00 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.440+0000 7fdf5dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.440+0000 7fdf5dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.440+0000 7fdf5dffb640 10 monclient(hunting): dump: 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:55.442 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:55.443 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:55.443 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1490776791 0 0) 0x7fdf6813c690 con 0x7fdf680b4e70 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 >> v1:172.21.15.7:6790/0 conn(0x7fdf680b1b00 legacy=0x7fdf6813acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 >> v1:172.21.15.183:6789/0 conn(0x7fdf680b44b0 legacy=0x7fdf68143b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 1 monclient: found mon.a 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf68127d00 con 0x7fdf680b4e70 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 10 monclient: _finish_auth 0 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf5dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 5 monclient: authenticate success, global_id 4542 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 10 monclient: _renew_subs 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:21:55.449 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf68139f50 con 0x7fdf680b4e70 2024-03-25T18:21:55.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 10 client.4542.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:21:55.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 10 monclient: _renew_subs 2024-03-25T18:21:55.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:21:55.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf6813cc90 con 0x7fdf680b4e70 2024-03-25T18:21:55.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.447+0000 7fdf6f895640 1 librados: init done 2024-03-25T18:21:55.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.448+0000 7fdf6f895640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:55.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.448+0000 7fdf6f895640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:55.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.448+0000 7fdf6f895640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:21:55.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.448+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf6813a250 con 0x7fdf680b4e70 2024-03-25T18:21:55.766 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.765+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdf60002000 con 0x7fdf680b4e70 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.765+0000 7fdf5dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.765+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fdf60004fb0 con 0x7fdf680b4e70 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.765+0000 7fdf5dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.765+0000 7fdf5dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.765+0000 7fdf5dffb640 10 monclient: dump: 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:21:55.767 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:21:55.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.765+0000 7fdf5f7fe640 4 set_mon_vals no callback set 2024-03-25T18:21:55.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.766+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fdf6001af80 con 0x7fdf680b4e70 2024-03-25T18:21:55.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.767+0000 7fdf6d631640 10 monclient: get_auth_request con 0x7fdf4c06e070 auth_method 0 2024-03-25T18:21:55.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.767+0000 7fdf5dffb640 10 client.4542.objecter ms_handle_connect 0x7fdf4c06e070 2024-03-25T18:21:55.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fdf60054ae0 con 0x7fdf680b4e70 2024-03-25T18:21:55.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 10 client.4542.objecter ms_dispatch 0x7fdf68000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:21:55.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 3 client.4542.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:21:55.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 3 client.4542.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:21:55.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 20 client.4542.objecter dump_active .. 0 homeless 2024-03-25T18:21:55.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdf60050420 con 0x7fdf680b4e70 2024-03-25T18:21:55.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:21:55.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:55.774+0000 7fdf5dffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:21:56.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.037+0000 7fdf6f895640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.037+0000 7fdf6f895640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.037+0000 7fdf6f895640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:21:56.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.037+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdf6804f730 con 0x7fdf680b4e70 2024-03-25T18:21:56.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.073+0000 7fdf5dffb640 1 -- 172.21.15.7:0/1820198213 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fdf60052520 con 0x7fdf680b4e70 2024-03-25T18:21:56.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.073+0000 7fdf5dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:21:56.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.073+0000 7fdf5dffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:21:56.074 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:21:56.075 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.075+0000 7fdf6f895640 10 librados: watch_flush enter 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 10 librados: watch_flush exit 2024-03-25T18:21:56.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 20 client.4542.objecter shutdown clearing up homeless session... 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 10 client.4542.objecter successfully canceled tick 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 >> v1:172.21.15.7:6800/1717564767 conn(0x7fdf4c06e070 legacy=0x7fdf4c070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 10 monclient: shutdown 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 >> v1:172.21.15.7:6789/0 conn(0x7fdf680b4e70 legacy=0x7fdf681e2120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 shutdown_connections 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 >> 172.21.15.7:0/1820198213 conn(0x7fdf680617a0 msgr2=0x7fdf680b2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 shutdown_connections 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 1 -- 172.21.15.7:0/1820198213 wait complete. 2024-03-25T18:21:56.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:21:56.076+0000 7fdf6f895640 1 librados: shutdown 2024-03-25T18:21:56.088 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:02.894 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:02.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.994+0000 7fe07ad7b640 10 monclient: get_monmap_and_config 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.994+0000 7fe07ad7b640 10 monclient: build_initial_monmap 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.994+0000 7fe07ad7b640 10 monclient: monmap: 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:02.995719+0000 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:02.995719+0000 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:02.996 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.995+0000 7fe07ad7b640 1 Processor -- start 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.995+0000 7fe07ad7b640 1 -- start start 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.995+0000 7fe07ad7b640 10 monclient: init 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0740b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0740b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe07413c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0740ab0e0 con 0x7fe0740b4670 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0740ab3c0 con 0x7fe0740b4ff0 2024-03-25T18:22:02.998 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0740a9f30 con 0x7fe07413c480 2024-03-25T18:22:02.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:02.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:02.996+0000 7fe07ad7b640 10 monclient(hunting): authenticate will time out at 7765.101358s 2024-03-25T18:22:03.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.016+0000 7fe073fff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fe0740b4670 0x7fe0741425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:56316/0 (socket says 172.21.15.7:56316) 2024-03-25T18:22:03.017 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.016+0000 7fe073fff640 1 -- 172.21.15.7:0/152333665 learned_addr learned my addr 172.21.15.7:0/152333665 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:03.027 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.026+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 697502434 0 0) 0x7fe0740ab0e0 con 0x7fe0740b4670 2024-03-25T18:22:03.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.026+0000 7fe0737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.026+0000 7fe0737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.026+0000 7fe0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.026+0000 7fe0737fe640 10 monclient(hunting): my global_id is 4545 2024-03-25T18:22:03.028 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.026+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe064003520 con 0x7fe0740b4670 2024-03-25T18:22:03.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.030+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1898625704 0 0) 0x7fe0740a9f30 con 0x7fe07413c480 2024-03-25T18:22:03.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.030+0000 7fe0737fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.030+0000 7fe0737fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.030+0000 7fe0737fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.030+0000 7fe0737fe640 10 monclient(hunting): my global_id is 4577 2024-03-25T18:22:03.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.030+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0740ab0e0 con 0x7fe07413c480 2024-03-25T18:22:03.035 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1899907663 0 0) 0x7fe064003520 con 0x7fe0740b4670 2024-03-25T18:22:03.035 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 >> v1:172.21.15.7:6790/0 conn(0x7fe07413c480 legacy=0x7fe07414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.035 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 >> v1:172.21.15.183:6789/0 conn(0x7fe0740b4ff0 legacy=0x7fe07413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.035 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:03.035 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 1 monclient: found mon.noname-a 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0740a9b70 con 0x7fe0740b4670 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe060001690 con 0x7fe0740b4670 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe07ad7b640 5 monclient: authenticate success, global_id 4545 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe07ad7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe0737fe640 10 monclient: dump: 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.036 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.037 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.037 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:03.037 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:03.037 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:03.037 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:03.037 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:03.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.034+0000 7fe07ad7b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe0600017f0 con 0x7fe0740b4670 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe0737fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe0737fe640 1 -- 172.21.15.7:0/152333665 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe060001c00 con 0x7fe0740b4670 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe0737fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe0737fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe0737fe640 10 monclient: dump: 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.059 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe07ad7b640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:03.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe07ad7b640 4 set_mon_vals no callback set 2024-03-25T18:22:03.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe07ad7b640 10 monclient: shutdown 2024-03-25T18:22:03.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe07ad7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:03.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.058+0000 7fe07ad7b640 1 -- 172.21.15.7:0/152333665 >> v1:172.21.15.7:6789/0 conn(0x7fe0740b4670 legacy=0x7fe0741425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.059+0000 7fe07ad7b640 1 -- 172.21.15.7:0/152333665 shutdown_connections 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.059+0000 7fe07ad7b640 1 -- 172.21.15.7:0/152333665 >> 172.21.15.7:0/152333665 conn(0x7fe0740617a0 msgr2=0x7fe074061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.059+0000 7fe07ad7b640 1 -- 172.21.15.7:0/152333665 shutdown_connections 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.059+0000 7fe07ad7b640 1 -- 172.21.15.7:0/152333665 wait complete. 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.059+0000 7fe07ad7b640 10 monclient: build_initial_monmap 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.059+0000 7fe07ad7b640 10 monclient: monmap: 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:03.063 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.060+0000 7fe07ad7b640 1 librados: starting msgr at 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.060+0000 7fe07ad7b640 1 librados: starting objecter 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.060+0000 7fe07ad7b640 1 Processor -- start 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.060+0000 7fe07ad7b640 1 -- start start 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.060+0000 7fe07ad7b640 1 librados: setting wanted keys 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.060+0000 7fe07ad7b640 1 librados: calling monclient init 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.060+0000 7fe07ad7b640 10 monclient: init 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 10 monclient(hunting): picked mon.noname-a con 0x7fe0740b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 10 monclient(hunting): picked mon.noname-c con 0x7fe0740b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 10 monclient(hunting): picked mon.noname-b con 0x7fe07413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0740aedf0 con 0x7fe0740b4670 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0740b0220 con 0x7fe07413c480 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0740b0500 con 0x7fe0740b4ff0 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:03.064 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.061+0000 7fe07ad7b640 10 monclient(hunting): authenticate will time out at 7765.165798s 2024-03-25T18:22:03.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.073+0000 7fe079318640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe07413c480 0x7fe0741e2170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:34844/0 (socket says 172.21.15.7:34844) 2024-03-25T18:22:03.074 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.073+0000 7fe079318640 1 -- 172.21.15.7:0/527096967 learned_addr learned my addr 172.21.15.7:0/527096967 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:03.076 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.075+0000 7fe0717fa640 10 client.?.objecter ms_handle_connect 0x7fe07413c480 2024-03-25T18:22:03.076 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.075+0000 7fe0717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.076+0000 7fe0717fa640 10 client.?.objecter ms_handle_connect 0x7fe0740b4ff0 2024-03-25T18:22:03.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.076+0000 7fe0717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.081+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1544928242 0 0) 0x7fe0740b0220 con 0x7fe07413c480 2024-03-25T18:22:03.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.081+0000 7fe0717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.081+0000 7fe0717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.081+0000 7fe0717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.081+0000 7fe0717fa640 10 monclient(hunting): my global_id is 4549 2024-03-25T18:22:03.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.081+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe058004ee0 con 0x7fe07413c480 2024-03-25T18:22:03.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.082+0000 7fe0717fa640 10 client.?.objecter ms_handle_connect 0x7fe0740b4670 2024-03-25T18:22:03.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.082+0000 7fe0717fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:03.086 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.085+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3623034070 0 0) 0x7fe0740b0500 con 0x7fe0740b4ff0 2024-03-25T18:22:03.086 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.085+0000 7fe0717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.086 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.085+0000 7fe0717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.085+0000 7fe0717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.085+0000 7fe0717fa640 10 monclient(hunting): my global_id is 4580 2024-03-25T18:22:03.087 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.085+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0740b0220 con 0x7fe0740b4ff0 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.096+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2452596383 0 0) 0x7fe058004ee0 con 0x7fe07413c480 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.096+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0740b0500 con 0x7fe07413c480 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.096+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe06c001c90 con 0x7fe07413c480 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.096+0000 7fe0717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.096+0000 7fe0717fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.096+0000 7fe0717fa640 10 monclient(hunting): dump: 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.098 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.096+0000 7fe0717fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.097+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321960009 0 0) 0x7fe0740aedf0 con 0x7fe0740b4670 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.097+0000 7fe0717fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.097+0000 7fe0717fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.097+0000 7fe0717fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.097+0000 7fe0717fa640 10 monclient(hunting): my global_id is 4548 2024-03-25T18:22:03.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.097+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe058004ee0 con 0x7fe0740b4670 2024-03-25T18:22:03.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.108+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2645767479 0 0) 0x7fe0740b0220 con 0x7fe0740b4ff0 2024-03-25T18:22:03.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.108+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe0740aedf0 con 0x7fe0740b4ff0 2024-03-25T18:22:03.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.108+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe060001b40 con 0x7fe0740b4ff0 2024-03-25T18:22:03.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.108+0000 7fe0717fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.108+0000 7fe0717fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:03.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.108+0000 7fe0717fa640 10 monclient(hunting): dump: 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1131750737 0 0) 0x7fe0740b0500 con 0x7fe07413c480 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 >> v1:172.21.15.7:6790/0 conn(0x7fe0740b4ff0 legacy=0x7fe0740b0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 >> v1:172.21.15.7:6789/0 conn(0x7fe0740b4670 legacy=0x7fe0740ae380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:03.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 1 monclient: found mon.b 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0740b07e0 con 0x7fe07413c480 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe0717fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 5 monclient: authenticate success, global_id 4549 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 10 monclient: _renew_subs 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0740b1310 con 0x7fe07413c480 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 10 monclient: _renew_subs 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0740af460 con 0x7fe07413c480 2024-03-25T18:22:03.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.109+0000 7fe07ad7b640 1 librados: init done 2024-03-25T18:22:03.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.110+0000 7fe07ad7b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.110+0000 7fe07ad7b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.110+0000 7fe07ad7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:03.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.110+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe07406dbd0 con 0x7fe07413c480 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.384+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe06c003990 con 0x7fe07413c480 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.384+0000 7fe0717fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.384+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe06c003d20 con 0x7fe07413c480 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.384+0000 7fe0717fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.384+0000 7fe0717fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.384+0000 7fe0717fa640 10 monclient: dump: 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.386 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.384+0000 7fe072ffd640 4 set_mon_vals no callback set 2024-03-25T18:22:03.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.385+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fe06c003f40 con 0x7fe07413c480 2024-03-25T18:22:03.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.386+0000 7fe078b17640 10 monclient: get_auth_request con 0x7fe05806e000 auth_method 0 2024-03-25T18:22:03.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.386+0000 7fe0717fa640 10 client.4549.objecter ms_handle_connect 0x7fe05806e000 2024-03-25T18:22:03.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.390+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fe06c054aa0 con 0x7fe07413c480 2024-03-25T18:22:03.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.390+0000 7fe0717fa640 10 client.4549.objecter ms_dispatch 0x7fe074000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:03.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.390+0000 7fe0717fa640 3 client.4549.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:03.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.390+0000 7fe0717fa640 3 client.4549.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:03.391 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.390+0000 7fe0717fa640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-25T18:22:03.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.392+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe06c0505b0 con 0x7fe07413c480 2024-03-25T18:22:03.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.392+0000 7fe0717fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:03.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.392+0000 7fe0717fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:03.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.645+0000 7fe07ad7b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:03.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.645+0000 7fe07ad7b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:03.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.645+0000 7fe07ad7b640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:03.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.645+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe0740688c0 con 0x7fe07413c480 2024-03-25T18:22:03.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.667+0000 7fe0717fa640 1 -- 172.21.15.7:0/527096967 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fe06c0524e0 con 0x7fe07413c480 2024-03-25T18:22:03.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.667+0000 7fe0717fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:03.669 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.667+0000 7fe0717fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:03.669 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:03.669 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 10 librados: watch_flush enter 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 10 librados: watch_flush exit 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 10 client.4549.objecter successfully canceled tick 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 >> v1:172.21.15.7:6800/1717564767 conn(0x7fe05806e000 legacy=0x7fe0580704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 10 monclient: shutdown 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 >> v1:172.21.15.183:6789/0 conn(0x7fe07413c480 legacy=0x7fe0741e2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 shutdown_connections 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 >> 172.21.15.7:0/527096967 conn(0x7fe0740617a0 msgr2=0x7fe0740b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.670+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 shutdown_connections 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.671+0000 7fe07ad7b640 1 -- 172.21.15.7:0/527096967 wait complete. 2024-03-25T18:22:03.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:03.671+0000 7fe07ad7b640 1 librados: shutdown 2024-03-25T18:22:03.684 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:08.942 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:15.579 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:08.948 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:15.585 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:08.986 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:15.623 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:08.999 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:15.636 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:09.005 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:15.642 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:09.021 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:15.647 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:09.026 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:15.652 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:09.049 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:15.687 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:09.066 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:15.692 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:10.491 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.603+0000 7f8966a6e640 10 monclient: get_monmap_and_config 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.603+0000 7f8966a6e640 10 monclient: build_initial_monmap 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.603+0000 7f8966a6e640 10 monclient: monmap: 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:10.604468+0000 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:10.604468+0000 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:10.604 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:10.605 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:10.605 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:10.605 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:10.605 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:10.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.604+0000 7f8966a6e640 1 Processor -- start 2024-03-25T18:22:10.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.604+0000 7f8966a6e640 1 -- start start 2024-03-25T18:22:10.605 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.604+0000 7f8966a6e640 10 monclient: init 2024-03-25T18:22:10.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:10.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:10.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 10 monclient(hunting): picked mon.noname-a con 0x7f89600b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:10.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 10 monclient(hunting): picked mon.noname-b con 0x7f89600b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:10.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 10 monclient(hunting): picked mon.noname-c con 0x7f896013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:10.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600ab0e0 con 0x7f89600b4ff0 2024-03-25T18:22:10.606 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600ab3c0 con 0x7f896013c480 2024-03-25T18:22:10.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600a9f30 con 0x7f89600b4670 2024-03-25T18:22:10.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:10.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.605+0000 7f8966a6e640 10 monclient(hunting): authenticate will time out at 7772.710030s 2024-03-25T18:22:10.628 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.627+0000 7f896480a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f89600b4ff0 0x7f896013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:32812/0 (socket says 172.21.15.7:32812) 2024-03-25T18:22:10.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.627+0000 7f896480a640 1 -- 172.21.15.7:0/2597545395 learned_addr learned my addr 172.21.15.7:0/2597545395 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:10.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.638+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1313791259 0 0) 0x7f89600a9f30 con 0x7f89600b4670 2024-03-25T18:22:10.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.638+0000 7f895f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:10.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.638+0000 7f895f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:10.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.638+0000 7f895f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:10.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.638+0000 7f895f7fe640 10 monclient(hunting): my global_id is 4583 2024-03-25T18:22:10.639 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.638+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8950003520 con 0x7f89600b4670 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.647+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1430626294 0 0) 0x7f8950003520 con 0x7f89600b4670 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.647+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 >> v1:172.21.15.183:6789/0 conn(0x7f896013c480 legacy=0x7f896014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.647+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 >> v1:172.21.15.7:6789/0 conn(0x7f89600b4ff0 legacy=0x7f896013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.647+0000 7f895f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.647+0000 7f895f7fe640 1 monclient: found mon.noname-b 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:22:10.649 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89600a9b70 con 0x7f89600b4670 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f894c001670 con 0x7f89600b4670 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f895f7fe640 10 monclient: dump: 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:10.650 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f8966a6e640 5 monclient: authenticate success, global_id 4583 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.648+0000 7f8966a6e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f894c0017d0 con 0x7f89600b4670 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f895f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f895f7fe640 1 -- 172.21.15.7:0/2597545395 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f894c001c20 con 0x7f89600b4670 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f8966a6e640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:10.651 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f8966a6e640 4 set_mon_vals no callback set 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f895f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f895f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.650+0000 7f895f7fe640 10 monclient: dump: 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:10.652 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:10.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.651+0000 7f8966a6e640 10 monclient: shutdown 2024-03-25T18:22:10.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.651+0000 7f8966a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:10.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.651+0000 7f8966a6e640 1 -- 172.21.15.7:0/2597545395 >> v1:172.21.15.7:6790/0 conn(0x7f89600b4670 legacy=0x7f89601425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.651+0000 7f8966a6e640 1 -- 172.21.15.7:0/2597545395 shutdown_connections 2024-03-25T18:22:10.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.651+0000 7f8966a6e640 1 -- 172.21.15.7:0/2597545395 >> 172.21.15.7:0/2597545395 conn(0x7f89600617a0 msgr2=0x7f8960061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:10.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.651+0000 7f8966a6e640 1 -- 172.21.15.7:0/2597545395 shutdown_connections 2024-03-25T18:22:10.653 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.651+0000 7f8966a6e640 1 -- 172.21.15.7:0/2597545395 wait complete. 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.652+0000 7f8966a6e640 10 monclient: build_initial_monmap 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.652+0000 7f8966a6e640 10 monclient: monmap: 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:10.654 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:10.655 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.652+0000 7f8966a6e640 1 librados: starting msgr at 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.652+0000 7f8966a6e640 1 librados: starting objecter 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 1 Processor -- start 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 1 -- start start 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 1 librados: setting wanted keys 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 1 librados: calling monclient init 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient: init 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient(hunting): picked mon.noname-b con 0x7f89600b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient(hunting): picked mon.noname-c con 0x7f89600b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient(hunting): picked mon.noname-a con 0x7f896013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600b0210 con 0x7f896013c480 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600b04f0 con 0x7f89600b4670 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89600b07d0 con 0x7f89600b4ff0 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:10.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.653+0000 7f8966a6e640 10 monclient(hunting): authenticate will time out at 7772.758488s 2024-03-25T18:22:10.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.670+0000 7f895ffff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f89600b4ff0 0x7f89600ae4e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:45506/0 (socket says 172.21.15.7:45506) 2024-03-25T18:22:10.671 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.670+0000 7f895ffff640 1 -- 172.21.15.7:0/1655967638 learned_addr learned my addr 172.21.15.7:0/1655967638 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:10.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.671+0000 7f895d7fa640 10 client.?.objecter ms_handle_connect 0x7f89600b4ff0 2024-03-25T18:22:10.672 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.671+0000 7f895d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:10.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.681+0000 7f895d7fa640 10 client.?.objecter ms_handle_connect 0x7f89600b4670 2024-03-25T18:22:10.682 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.681+0000 7f895d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:10.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.691+0000 7f895d7fa640 10 client.?.objecter ms_handle_connect 0x7f896013c480 2024-03-25T18:22:10.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.691+0000 7f895d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:10.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.756+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 170711980 0 0) 0x7f89600b0210 con 0x7f896013c480 2024-03-25T18:22:10.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.756+0000 7f895d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:10.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.756+0000 7f895d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:10.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.756+0000 7f895d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:10.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.756+0000 7f895d7fa640 10 monclient(hunting): my global_id is 4554 2024-03-25T18:22:10.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.756+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8944004eb0 con 0x7f896013c480 2024-03-25T18:22:10.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.801+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1932637680 0 0) 0x7f8944004eb0 con 0x7f896013c480 2024-03-25T18:22:10.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.801+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f89600b0210 con 0x7f896013c480 2024-03-25T18:22:10.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.801+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8958001b90 con 0x7f896013c480 2024-03-25T18:22:10.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.801+0000 7f895d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:10.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.801+0000 7f895d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:10.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.801+0000 7f895d7fa640 10 monclient(hunting): dump: 2024-03-25T18:22:10.802 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:10.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.801+0000 7f895d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:10.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.823+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1184885402 0 0) 0x7f89600b04f0 con 0x7f89600b4670 2024-03-25T18:22:10.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.823+0000 7f895d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:10.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.823+0000 7f895d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:10.824 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.823+0000 7f895d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:10.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.823+0000 7f895d7fa640 10 monclient(hunting): my global_id is 4555 2024-03-25T18:22:10.825 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.823+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8944004eb0 con 0x7f89600b4670 2024-03-25T18:22:10.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.825+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3417690767 0 0) 0x7f89600b07d0 con 0x7f89600b4ff0 2024-03-25T18:22:10.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.825+0000 7f895d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:10.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.825+0000 7f895d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:10.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.825+0000 7f895d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:10.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.825+0000 7f895d7fa640 10 monclient(hunting): my global_id is 4586 2024-03-25T18:22:10.827 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.825+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f89600b04f0 con 0x7f89600b4ff0 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3387112972 0 0) 0x7f89600b0210 con 0x7f896013c480 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 >> v1:172.21.15.7:6790/0 conn(0x7f89600b4ff0 legacy=0x7f89600ae4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 >> v1:172.21.15.183:6789/0 conn(0x7f89600b4670 legacy=0x7f89600b0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 1 monclient: found mon.a 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89600b1670 con 0x7f896013c480 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f895d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 5 monclient: authenticate success, global_id 4554 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 10 monclient: _renew_subs 2024-03-25T18:22:10.831 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89601e3390 con 0x7f896013c480 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 10 client.4554.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 10 monclient: _renew_subs 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f89601e3700 con 0x7f896013c480 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.829+0000 7f8966a6e640 1 librados: init done 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.830+0000 7f8966a6e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.830+0000 7f8966a6e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.830+0000 7f8966a6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:22:10.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:10.830+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f896006dbd0 con 0x7f896013c480 2024-03-25T18:22:11.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.108+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f89580020c0 con 0x7f896013c480 2024-03-25T18:22:11.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.108+0000 7f895d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.108+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f8958003b30 con 0x7f896013c480 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.108+0000 7f895d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.108+0000 7f895d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.108+0000 7f895d7fa640 10 monclient: dump: 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:11.110 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:11.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.108+0000 7f895effd640 4 set_mon_vals no callback set 2024-03-25T18:22:11.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.109+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f8958003d10 con 0x7f896013c480 2024-03-25T18:22:11.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.110+0000 7f896480a640 10 monclient: get_auth_request con 0x7f894406de10 auth_method 0 2024-03-25T18:22:11.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.110+0000 7f895d7fa640 10 client.4554.objecter ms_handle_connect 0x7f894406de10 2024-03-25T18:22:11.131 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f89580538c0 con 0x7f896013c480 2024-03-25T18:22:11.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 10 client.4554.objecter ms_dispatch 0x7f8960000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:11.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 3 client.4554.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:11.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 3 client.4554.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:11.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 20 client.4554.objecter dump_active .. 0 homeless 2024-03-25T18:22:11.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f895804f200 con 0x7f896013c480 2024-03-25T18:22:11.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:11.132 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.130+0000 7f895d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:11.361 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.360+0000 7f8966a6e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:11.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.360+0000 7f8966a6e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:11.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.360+0000 7f8966a6e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:22:11.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.360+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89600688c0 con 0x7f896013c480 2024-03-25T18:22:11.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.405+0000 7f895d7fa640 1 -- 172.21.15.7:0/1655967638 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f8958051300 con 0x7f896013c480 2024-03-25T18:22:11.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.405+0000 7f895d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:11.406 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.405+0000 7f895d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:11.406 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:11.406 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:11.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 10 librados: watch_flush enter 2024-03-25T18:22:11.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 10 librados: watch_flush exit 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 20 client.4554.objecter shutdown clearing up homeless session... 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 10 client.4554.objecter successfully canceled tick 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 >> v1:172.21.15.7:6800/1717564767 conn(0x7f894406de10 legacy=0x7f89440702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 10 monclient: shutdown 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 >> v1:172.21.15.7:6789/0 conn(0x7f896013c480 legacy=0x7f89600aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 shutdown_connections 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 >> 172.21.15.7:0/1655967638 conn(0x7f89600617a0 msgr2=0x7f89600b7060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 shutdown_connections 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 1 -- 172.21.15.7:0/1655967638 wait complete. 2024-03-25T18:22:11.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:11.407+0000 7f8966a6e640 1 librados: shutdown 2024-03-25T18:22:11.421 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:18.227 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.342+0000 7f7efe3c6640 10 monclient: get_monmap_and_config 2024-03-25T18:22:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.342+0000 7f7efe3c6640 10 monclient: build_initial_monmap 2024-03-25T18:22:18.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.342+0000 7f7efe3c6640 10 monclient: monmap: 2024-03-25T18:22:18.343 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:18.343557+0000 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:18.343557+0000 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:18.344 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:18.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.343+0000 7f7efe3c6640 1 Processor -- start 2024-03-25T18:22:18.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.343+0000 7f7efe3c6640 1 -- start start 2024-03-25T18:22:18.345 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.343+0000 7f7efe3c6640 10 monclient: init 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f7ef80b1a70 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f7ef81405b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f7ef813d690 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef8072990 con 0x7f7ef81405b0 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef8072c70 con 0x7f7ef813d690 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef813dc30 con 0x7f7ef80b1a70 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:18.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.344+0000 7f7efe3c6640 10 monclient(hunting): authenticate will time out at 7780.449278s 2024-03-25T18:22:18.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.364+0000 7f7ef7fff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f7ef80b1a70 0x7f7ef813cf80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:45192/0 (socket says 172.21.15.7:45192) 2024-03-25T18:22:18.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.364+0000 7f7ef7fff640 1 -- 172.21.15.7:0/657074973 learned_addr learned my addr 172.21.15.7:0/657074973 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:18.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.371+0000 7f7ef7fff640 1 -- 172.21.15.7:0/657074973 >> v1:172.21.15.7:6790/0 conn(0x7f7ef80b1a70 legacy=0x7f7ef813cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-25T18:22:18.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.372+0000 7f7ef7fff640 1 -- 172.21.15.7:0/657074973 >> v1:172.21.15.7:6790/0 conn(0x7f7ef80b1a70 legacy=0x7f7ef813cf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:22:18.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.372+0000 7f7ef7fff640 1 --1- 172.21.15.7:0/657074973 >> v1:172.21.15.7:6790/0 conn(0x7f7ef80b1a70 0x7f7ef813cf80 :-1 s=OPENED pgs=57502 cs=1 l=1).handle_message read tag failed 2024-03-25T18:22:18.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.372+0000 7f7ef7fff640 1 --1- 172.21.15.7:0/657074973 >> v1:172.21.15.7:6790/0 conn(0x7f7ef80b1a70 0x7f7ef813cf80 :-1 s=OPENED pgs=57502 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:22:18.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.372+0000 7f7ef6ffd640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.7:6790/0 2024-03-25T18:22:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.381+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3478182792 0 0) 0x7f7ef8072c70 con 0x7f7ef813d690 2024-03-25T18:22:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.381+0000 7f7ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.381+0000 7f7ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.381+0000 7f7ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.381+0000 7f7ef6ffd640 10 monclient(hunting): my global_id is 4558 2024-03-25T18:22:18.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.381+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee0003520 con 0x7f7ef813d690 2024-03-25T18:22:18.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.395+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3040283884 0 0) 0x7f7ef8072990 con 0x7f7ef81405b0 2024-03-25T18:22:18.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.395+0000 7f7ef6ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.395+0000 7f7ef6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.395+0000 7f7ef6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.395+0000 7f7ef6ffd640 10 monclient(hunting): my global_id is 4557 2024-03-25T18:22:18.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.395+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef8072c70 con 0x7f7ef81405b0 2024-03-25T18:22:18.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2542026433 0 0) 0x7f7ee0003520 con 0x7f7ef813d690 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 >> v1:172.21.15.7:6790/0 conn(0x7f7ef80b1a70 legacy=0x7f7ef813cf80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 >> v1:172.21.15.7:6789/0 conn(0x7f7ef81405b0 legacy=0x7f7ef806ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 1 monclient: found mon.noname-c 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef8140b20 con 0x7f7ef813d690 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7ef0001a10 con 0x7f7ef813d690 2024-03-25T18:22:18.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7ef6ffd640 10 monclient: dump: 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7efe3c6640 5 monclient: authenticate success, global_id 4558 2024-03-25T18:22:18.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.396+0000 7f7efe3c6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:18.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ef0001b70 con 0x7f7ef813d690 2024-03-25T18:22:18.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7ef6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:18.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7ef6ffd640 1 -- 172.21.15.7:0/657074973 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7ef0001f80 con 0x7f7ef813d690 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7efe3c6640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7efe3c6640 4 set_mon_vals no callback set 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7ef6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7ef6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.403+0000 7f7ef6ffd640 10 monclient: dump: 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:18.405 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.404+0000 7f7efe3c6640 10 monclient: shutdown 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.404+0000 7f7efe3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.404+0000 7f7efe3c6640 1 -- 172.21.15.7:0/657074973 >> v1:172.21.15.183:6789/0 conn(0x7f7ef813d690 legacy=0x7f7ef814ba10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.404+0000 7f7efe3c6640 1 -- 172.21.15.7:0/657074973 shutdown_connections 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.404+0000 7f7efe3c6640 1 -- 172.21.15.7:0/657074973 >> 172.21.15.7:0/657074973 conn(0x7f7ef80b28f0 msgr2=0x7f7ef80b2d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.404+0000 7f7efe3c6640 1 -- 172.21.15.7:0/657074973 shutdown_connections 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.404+0000 7f7efe3c6640 1 -- 172.21.15.7:0/657074973 wait complete. 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.405+0000 7f7efe3c6640 10 monclient: build_initial_monmap 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.405+0000 7f7efe3c6640 10 monclient: monmap: 2024-03-25T18:22:18.407 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.405+0000 7f7efe3c6640 1 librados: starting msgr at 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.405+0000 7f7efe3c6640 1 librados: starting objecter 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.405+0000 7f7efe3c6640 1 Processor -- start 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.406+0000 7f7efe3c6640 1 -- start start 2024-03-25T18:22:18.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.406+0000 7f7efe3c6640 1 librados: setting wanted keys 2024-03-25T18:22:18.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.406+0000 7f7efe3c6640 1 librados: calling monclient init 2024-03-25T18:22:18.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.406+0000 7f7efe3c6640 10 monclient: init 2024-03-25T18:22:18.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.406+0000 7f7efe3c6640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:18.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.406+0000 7f7efe3c6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:18.409 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.406+0000 7f7efe3c6640 10 monclient(hunting): picked mon.noname-a con 0x7f7ef80b1a70 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:18.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.407+0000 7f7efe3c6640 10 monclient(hunting): picked mon.noname-c con 0x7f7ef813d690 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:18.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.407+0000 7f7efe3c6640 10 monclient(hunting): picked mon.noname-b con 0x7f7ef81405b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:18.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.407+0000 7f7efe3c6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef80ad520 con 0x7f7ef80b1a70 2024-03-25T18:22:18.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.407+0000 7f7efe3c6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef80ab560 con 0x7f7ef81405b0 2024-03-25T18:22:18.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.407+0000 7f7efe3c6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ef8143880 con 0x7f7ef813d690 2024-03-25T18:22:18.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.407+0000 7f7efe3c6640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:18.410 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.407+0000 7f7efe3c6640 10 monclient(hunting): authenticate will time out at 7780.511697s 2024-03-25T18:22:18.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.418+0000 7f7efc963640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f7ef81405b0 0x7f7ef80aae50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:50954/0 (socket says 172.21.15.7:50954) 2024-03-25T18:22:18.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.419+0000 7f7efc963640 1 -- 172.21.15.7:0/1840583699 learned_addr learned my addr 172.21.15.7:0/1840583699 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:18.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.419+0000 7f7ef4ff9640 10 client.?.objecter ms_handle_connect 0x7f7ef81405b0 2024-03-25T18:22:18.420 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.419+0000 7f7ef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:18.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.427+0000 7f7ef4ff9640 10 client.?.objecter ms_handle_connect 0x7f7ef813d690 2024-03-25T18:22:18.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.427+0000 7f7ef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:18.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.435+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3919104632 0 0) 0x7f7ef80ab560 con 0x7f7ef81405b0 2024-03-25T18:22:18.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.435+0000 7f7ef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.435+0000 7f7ef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.435+0000 7f7ef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.435+0000 7f7ef4ff9640 10 monclient(hunting): my global_id is 4561 2024-03-25T18:22:18.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.435+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee4004eb0 con 0x7f7ef81405b0 2024-03-25T18:22:18.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.441+0000 7f7ef4ff9640 10 client.?.objecter ms_handle_connect 0x7f7ef80b1a70 2024-03-25T18:22:18.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.441+0000 7f7ef4ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:18.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.444+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1872202774 0 0) 0x7f7ef8143880 con 0x7f7ef813d690 2024-03-25T18:22:18.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.444+0000 7f7ef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.444+0000 7f7ef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.444+0000 7f7ef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.444+0000 7f7ef4ff9640 10 monclient(hunting): my global_id is 4592 2024-03-25T18:22:18.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.444+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ef80ab560 con 0x7f7ef813d690 2024-03-25T18:22:18.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.446+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 662097182 0 0) 0x7f7ee4004eb0 con 0x7f7ef81405b0 2024-03-25T18:22:18.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.446+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7ef8143880 con 0x7f7ef81405b0 2024-03-25T18:22:18.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.446+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7ef0001ec0 con 0x7f7ef81405b0 2024-03-25T18:22:18.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.446+0000 7f7ef4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.446+0000 7f7ef4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:18.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.446+0000 7f7ef4ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:18.448 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.446+0000 7f7ef4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:18.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.455+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 974641799 0 0) 0x7f7ef80ad520 con 0x7f7ef80b1a70 2024-03-25T18:22:18.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.455+0000 7f7ef4ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:18.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.455+0000 7f7ef4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:18.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.455+0000 7f7ef4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:18.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.455+0000 7f7ef4ff9640 10 monclient(hunting): my global_id is 4560 2024-03-25T18:22:18.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.455+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7ee4004eb0 con 0x7f7ef80b1a70 2024-03-25T18:22:18.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2157202031 0 0) 0x7f7ef8143880 con 0x7f7ef81405b0 2024-03-25T18:22:18.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 >> v1:172.21.15.7:6790/0 conn(0x7f7ef813d690 legacy=0x7f7ef80abe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 >> v1:172.21.15.7:6789/0 conn(0x7f7ef80b1a70 legacy=0x7f7ef80a9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 1 monclient: found mon.b 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef8143a10 con 0x7f7ef81405b0 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7ef4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 5 monclient: authenticate success, global_id 4561 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 10 monclient: _renew_subs 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ef8138160 con 0x7f7ef81405b0 2024-03-25T18:22:18.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 10 client.4561.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:18.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 10 monclient: _renew_subs 2024-03-25T18:22:18.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:18.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ef8138690 con 0x7f7ef81405b0 2024-03-25T18:22:18.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.462+0000 7f7efe3c6640 1 librados: init done 2024-03-25T18:22:18.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.463+0000 7f7efe3c6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:18.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.463+0000 7f7efe3c6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:18.465 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.463+0000 7f7efe3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:18.466 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.463+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ef8143c00 con 0x7f7ef81405b0 2024-03-25T18:22:18.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7ef0003c20 con 0x7f7ef81405b0 2024-03-25T18:22:18.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:18.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f7ef00015d0 con 0x7f7ef81405b0 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef4ff9640 10 monclient: dump: 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:18.743 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:18.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef67fc640 4 set_mon_vals no callback set 2024-03-25T18:22:18.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.741+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f7ef00040f0 con 0x7f7ef81405b0 2024-03-25T18:22:18.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.742+0000 7f7ef7fff640 10 monclient: get_auth_request con 0x7f7ee406de00 auth_method 0 2024-03-25T18:22:18.744 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.742+0000 7f7ef4ff9640 10 client.4561.objecter ms_handle_connect 0x7f7ee406de00 2024-03-25T18:22:18.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.746+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f7ef0054be0 con 0x7f7ef81405b0 2024-03-25T18:22:18.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.746+0000 7f7ef4ff9640 10 client.4561.objecter ms_dispatch 0x7f7ef8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:18.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.746+0000 7f7ef4ff9640 3 client.4561.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:18.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.746+0000 7f7ef4ff9640 3 client.4561.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:18.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.746+0000 7f7ef4ff9640 20 client.4561.objecter dump_active .. 0 homeless 2024-03-25T18:22:18.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.747+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7ef00506f0 con 0x7f7ef81405b0 2024-03-25T18:22:18.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.747+0000 7f7ef4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:18.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:18.747+0000 7f7ef4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:19.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.025+0000 7f7efe3c6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.025+0000 7f7efe3c6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.025+0000 7f7efe3c6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:19.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.025+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7ef80688c0 con 0x7f7ef81405b0 2024-03-25T18:22:19.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.056+0000 7f7ef4ff9640 1 -- 172.21.15.7:0/1840583699 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f7ef0052620 con 0x7f7ef81405b0 2024-03-25T18:22:19.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.056+0000 7f7ef4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:19.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.056+0000 7f7ef4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:19.057 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:19.057 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:19.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 10 librados: watch_flush enter 2024-03-25T18:22:19.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 10 librados: watch_flush exit 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 20 client.4561.objecter shutdown clearing up homeless session... 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 10 client.4561.objecter successfully canceled tick 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 >> v1:172.21.15.7:6800/1717564767 conn(0x7f7ee406de00 legacy=0x7f7ee40702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 10 monclient: shutdown 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 >> v1:172.21.15.183:6789/0 conn(0x7f7ef81405b0 legacy=0x7f7ef80aae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 shutdown_connections 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 >> 172.21.15.7:0/1840583699 conn(0x7f7ef80b28f0 msgr2=0x7f7ef813cd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 shutdown_connections 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 1 -- 172.21.15.7:0/1840583699 wait complete. 2024-03-25T18:22:19.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:19.059+0000 7f7efe3c6640 1 librados: shutdown 2024-03-25T18:22:19.074 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:25.880 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:26.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:25.999+0000 7effcd179640 10 monclient: get_monmap_and_config 2024-03-25T18:22:26.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:25.999+0000 7effcd179640 10 monclient: build_initial_monmap 2024-03-25T18:22:26.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:25.999+0000 7effcd179640 10 monclient: monmap: 2024-03-25T18:22:26.000 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:26.000671+0000 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:26.000671+0000 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:26.001 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.000+0000 7effcd179640 1 Processor -- start 2024-03-25T18:22:26.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.000+0000 7effcd179640 1 -- start start 2024-03-25T18:22:26.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.000+0000 7effcd179640 10 monclient: init 2024-03-25T18:22:26.002 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.000+0000 7effcd179640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 10 monclient(hunting): picked mon.noname-b con 0x7effc80b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 10 monclient(hunting): picked mon.noname-c con 0x7effc80b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 10 monclient(hunting): picked mon.noname-a con 0x7effc80b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc80a96f0 con 0x7effc80b1b00 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc80ab0e0 con 0x7effc80b44b0 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc8145f70 con 0x7effc80b4e70 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:26.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.001+0000 7effcd179640 10 monclient(hunting): authenticate will time out at 7788.105743s 2024-03-25T18:22:26.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.015+0000 7effc6d9d640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7effc80b4e70 0x7effc813f940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:40086/0 (socket says 172.21.15.7:40086) 2024-03-25T18:22:26.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.015+0000 7effc6d9d640 1 -- 172.21.15.7:0/2681280761 learned_addr learned my addr 172.21.15.7:0/2681280761 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.023+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656023988 0 0) 0x7effc80a96f0 con 0x7effc80b1b00 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.023+0000 7effc5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.023+0000 7effc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.023+0000 7effc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.023+0000 7effc5d9b640 10 monclient(hunting): my global_id is 4563 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.024+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb0003520 con 0x7effc80b1b00 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.024+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2896708666 0 0) 0x7effc8145f70 con 0x7effc80b4e70 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.024+0000 7effc5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.024+0000 7effc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.024+0000 7effc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.024+0000 7effc5d9b640 10 monclient(hunting): my global_id is 4595 2024-03-25T18:22:26.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.024+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc80a96f0 con 0x7effc80b4e70 2024-03-25T18:22:26.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.030+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546867373 0 0) 0x7effc80ab0e0 con 0x7effc80b44b0 2024-03-25T18:22:26.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.030+0000 7effc5d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.030+0000 7effc5d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.030+0000 7effc5d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.030+0000 7effc5d9b640 10 monclient(hunting): my global_id is 4564 2024-03-25T18:22:26.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.030+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8145f70 con 0x7effc80b44b0 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3975651529 0 0) 0x7effb0003520 con 0x7effc80b1b00 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 >> v1:172.21.15.7:6790/0 conn(0x7effc80b4e70 legacy=0x7effc813f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 >> v1:172.21.15.183:6789/0 conn(0x7effc80b44b0 legacy=0x7effc8145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 1 monclient: found mon.noname-a 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:22:26.036 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc80a9b80 con 0x7effc80b1b00 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 10 monclient: _finish_auth 0 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7effc0001a10 con 0x7effc80b1b00 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effcd179640 5 monclient: authenticate success, global_id 4563 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effcd179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effc5d9b640 10 monclient: dump: 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.037 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:26.038 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:26.038 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:26.038 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:26.038 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.035+0000 7effcd179640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:26.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effc0001b70 con 0x7effc80b1b00 2024-03-25T18:22:26.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effc5d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effc5d9b640 1 -- 172.21.15.7:0/2681280761 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7effc0001f80 con 0x7effc80b1b00 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effcd179640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effcd179640 4 set_mon_vals no callback set 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effc5d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effc5d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.049+0000 7effc5d9b640 10 monclient: dump: 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:26.051 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.050+0000 7effcd179640 10 monclient: shutdown 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.050+0000 7effcd179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.050+0000 7effcd179640 1 -- 172.21.15.7:0/2681280761 >> v1:172.21.15.7:6789/0 conn(0x7effc80b1b00 legacy=0x7effc814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.050+0000 7effcd179640 1 -- 172.21.15.7:0/2681280761 shutdown_connections 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.050+0000 7effcd179640 1 -- 172.21.15.7:0/2681280761 >> 172.21.15.7:0/2681280761 conn(0x7effc80617a0 msgr2=0x7effc8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.050+0000 7effcd179640 1 -- 172.21.15.7:0/2681280761 shutdown_connections 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.050+0000 7effcd179640 1 -- 172.21.15.7:0/2681280761 wait complete. 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.051+0000 7effcd179640 10 monclient: build_initial_monmap 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.051+0000 7effcd179640 10 monclient: monmap: 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:26.053 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:26.054 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:26.054 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:26.054 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:26.054 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.054 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 1 librados: starting msgr at 2024-03-25T18:22:26.054 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 1 librados: starting objecter 2024-03-25T18:22:26.054 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 1 Processor -- start 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 1 -- start start 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 1 librados: setting wanted keys 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 1 librados: calling monclient init 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 10 monclient: init 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.052+0000 7effcd179640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 10 monclient(hunting): picked mon.noname-a con 0x7effc80b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 10 monclient(hunting): picked mon.noname-b con 0x7effc80b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:26.055 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 10 monclient(hunting): picked mon.noname-c con 0x7effc80b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:26.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc8138ed0 con 0x7effc80b1b00 2024-03-25T18:22:26.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc8135b30 con 0x7effc80b44b0 2024-03-25T18:22:26.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effc8135cc0 con 0x7effc80b4e70 2024-03-25T18:22:26.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:26.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.053+0000 7effcd179640 10 monclient(hunting): authenticate will time out at 7788.157806s 2024-03-25T18:22:26.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.055+0000 7effc659c640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7effc80b44b0 0x7effc8133f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:42014/0 (socket says 172.21.15.7:42014) 2024-03-25T18:22:26.056 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.055+0000 7effc659c640 1 -- 172.21.15.7:0/4088656720 learned_addr learned my addr 172.21.15.7:0/4088656720 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:26.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.056+0000 7effaf7fe640 10 client.?.objecter ms_handle_connect 0x7effc80b44b0 2024-03-25T18:22:26.057 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.056+0000 7effaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.072+0000 7effaf7fe640 10 client.?.objecter ms_handle_connect 0x7effc80b1b00 2024-03-25T18:22:26.073 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.072+0000 7effaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.076 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.075+0000 7effaf7fe640 10 client.?.objecter ms_handle_connect 0x7effc80b4e70 2024-03-25T18:22:26.076 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.075+0000 7effaf7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:26.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.077+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2011980562 0 0) 0x7effc8135b30 con 0x7effc80b44b0 2024-03-25T18:22:26.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.077+0000 7effaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.077+0000 7effaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.078 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.077+0000 7effaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.077+0000 7effaf7fe640 10 monclient(hunting): my global_id is 4567 2024-03-25T18:22:26.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.077+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effb4004eb0 con 0x7effc80b44b0 2024-03-25T18:22:26.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.092+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2796766390 0 0) 0x7effc8135cc0 con 0x7effc80b4e70 2024-03-25T18:22:26.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.092+0000 7effaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.092+0000 7effaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.092+0000 7effaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.092+0000 7effaf7fe640 10 monclient(hunting): my global_id is 4598 2024-03-25T18:22:26.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.092+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8135b30 con 0x7effc80b4e70 2024-03-25T18:22:26.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.094+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313177248 0 0) 0x7effc8138ed0 con 0x7effc80b1b00 2024-03-25T18:22:26.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.094+0000 7effaf7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:26.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.094+0000 7effaf7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:26.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.094+0000 7effaf7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:26.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.094+0000 7effaf7fe640 10 monclient(hunting): my global_id is 4566 2024-03-25T18:22:26.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.094+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effc8135cc0 con 0x7effc80b1b00 2024-03-25T18:22:26.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 400647259 0 0) 0x7effb4004eb0 con 0x7effc80b44b0 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effc8138ed0 con 0x7effc80b44b0 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7effbc0019d0 con 0x7effc80b44b0 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 10 monclient(hunting): dump: 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:26.106 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3998532580 0 0) 0x7effc8135b30 con 0x7effc80b4e70 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effb4004eb0 con 0x7effc80b4e70 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7effc0001e80 con 0x7effc80b4e70 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.104+0000 7effaf7fe640 10 monclient(hunting): dump: 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:26.107 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:26.108 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.110+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3614212380 0 0) 0x7effc8135cc0 con 0x7effc80b1b00 2024-03-25T18:22:26.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.110+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7effc8135b30 con 0x7effc80b1b00 2024-03-25T18:22:26.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.110+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7effb80018d0 con 0x7effc80b1b00 2024-03-25T18:22:26.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.110+0000 7effaf7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.110+0000 7effaf7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:26.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.110+0000 7effaf7fe640 10 monclient(hunting): dump: 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:26.112 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 428517120 0 0) 0x7effb4004eb0 con 0x7effc80b4e70 2024-03-25T18:22:26.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 >> v1:172.21.15.183:6789/0 conn(0x7effc80b44b0 legacy=0x7effc8133f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 >> v1:172.21.15.7:6789/0 conn(0x7effc80b1b00 legacy=0x7effc8139360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.118 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 1 monclient: found mon.c 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effc8139a70 con 0x7effc80b4e70 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effaf7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 5 monclient: authenticate success, global_id 4598 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 10 monclient: _renew_subs 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effc8139c00 con 0x7effc80b4e70 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 10 monclient: _renew_subs 2024-03-25T18:22:26.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:26.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7effc81dbd10 con 0x7effc80b4e70 2024-03-25T18:22:26.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.117+0000 7effcd179640 1 librados: init done 2024-03-25T18:22:26.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.118+0000 7effcd179640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.118+0000 7effcd179640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.118+0000 7effcd179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:26.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.118+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effc8138060 con 0x7effc80b4e70 2024-03-25T18:22:26.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.433+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7effc0001fe0 con 0x7effc80b4e70 2024-03-25T18:22:26.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.433+0000 7effaf7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:26.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.433+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7effc0003c20 con 0x7effc80b4e70 2024-03-25T18:22:26.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.433+0000 7effaf7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.433+0000 7effaf7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.433+0000 7effaf7fe640 10 monclient: dump: 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:26.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.433+0000 7effc559a640 4 set_mon_vals no callback set 2024-03-25T18:22:26.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.434+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7effc0019bf0 con 0x7effc80b4e70 2024-03-25T18:22:26.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.435+0000 7effc6d9d640 10 monclient: get_auth_request con 0x7effb406e110 auth_method 0 2024-03-25T18:22:26.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.435+0000 7effaf7fe640 10 client.4598.objecter ms_handle_connect 0x7effb406e110 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.439+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7effc00536f0 con 0x7effc80b4e70 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.439+0000 7effaf7fe640 10 client.4598.objecter ms_dispatch 0x7effc8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.439+0000 7effaf7fe640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.439+0000 7effaf7fe640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.439+0000 7effaf7fe640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.439+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7effc004f030 con 0x7effc80b4e70 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.440+0000 7effaf7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:26.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.440+0000 7effaf7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:26.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.695+0000 7effcd179640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:26.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.695+0000 7effcd179640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:26.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.695+0000 7effcd179640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:26.696 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.695+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effc804f730 con 0x7effc80b4e70 2024-03-25T18:22:26.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.864+0000 7effaf7fe640 1 -- 172.21.15.7:0/4088656720 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7effc0051130 con 0x7effc80b4e70 2024-03-25T18:22:26.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.864+0000 7effaf7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:26.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.864+0000 7effaf7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:26.866 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:26.866 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 10 librados: watch_flush enter 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 10 librados: watch_flush exit 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 10 client.4598.objecter successfully canceled tick 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 >> v1:172.21.15.7:6800/1717564767 conn(0x7effb406e110 legacy=0x7effb40705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 10 monclient: shutdown 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 >> v1:172.21.15.7:6790/0 conn(0x7effc80b4e70 legacy=0x7effc81e3220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 shutdown_connections 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 >> 172.21.15.7:0/4088656720 conn(0x7effc80617a0 msgr2=0x7effc80b30a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 shutdown_connections 2024-03-25T18:22:26.868 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.866+0000 7effcd179640 1 -- 172.21.15.7:0/4088656720 wait complete. 2024-03-25T18:22:26.869 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:26.867+0000 7effcd179640 1 librados: shutdown 2024-03-25T18:22:26.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:33.688 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:33.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.802+0000 7f4866b3e640 10 monclient: get_monmap_and_config 2024-03-25T18:22:33.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.802+0000 7f4866b3e640 10 monclient: build_initial_monmap 2024-03-25T18:22:33.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.802+0000 7f4866b3e640 10 monclient: monmap: 2024-03-25T18:22:33.803 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:33.803 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:33.803 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:33.803402+0000 2024-03-25T18:22:33.804 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:33.803402+0000 2024-03-25T18:22:33.804 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:33.804 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:33.804 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:33.804 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:33.804 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:33.804 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:33.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 1 Processor -- start 2024-03-25T18:22:33.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 1 -- start start 2024-03-25T18:22:33.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 10 monclient: init 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f48600b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f48600b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 10 monclient(hunting): picked mon.noname-b con 0x7f486013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.803+0000 7f4866b3e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48600ab0e0 con 0x7f48600b4670 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.804+0000 7f4866b3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48600ab3c0 con 0x7f48600b4ff0 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.804+0000 7f4866b3e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48600a9f30 con 0x7f486013c480 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.804+0000 7f4866b3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:33.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.804+0000 7f4866b3e640 10 monclient(hunting): authenticate will time out at 7795.908671s 2024-03-25T18:22:33.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.820+0000 7f48650db640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f486013c480 0x7f486014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:57842/0 (socket says 172.21.15.7:57842) 2024-03-25T18:22:33.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.820+0000 7f48650db640 1 -- 172.21.15.7:0/3351749666 learned_addr learned my addr 172.21.15.7:0/3351749666 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:33.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.831+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1711925863 0 0) 0x7f48600ab0e0 con 0x7f48600b4670 2024-03-25T18:22:33.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.831+0000 7f485f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.831+0000 7f485f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.831+0000 7f485f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.831+0000 7f485f7fe640 10 monclient(hunting): my global_id is 4569 2024-03-25T18:22:33.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.831+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4850003520 con 0x7f48600b4670 2024-03-25T18:22:33.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.838+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3195430604 0 0) 0x7f48600ab3c0 con 0x7f48600b4ff0 2024-03-25T18:22:33.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.838+0000 7f485f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.838+0000 7f485f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.838+0000 7f485f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.838+0000 7f485f7fe640 10 monclient(hunting): my global_id is 4570 2024-03-25T18:22:33.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.838+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48600ab0e0 con 0x7f48600b4ff0 2024-03-25T18:22:33.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.844+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837044668 0 0) 0x7f48600a9f30 con 0x7f486013c480 2024-03-25T18:22:33.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.844+0000 7f485f7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.844+0000 7f485f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.844+0000 7f485f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.844+0000 7f485f7fe640 10 monclient(hunting): my global_id is 4601 2024-03-25T18:22:33.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.844+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48600ab3c0 con 0x7f486013c480 2024-03-25T18:22:33.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1536484193 0 0) 0x7f4850003520 con 0x7f48600b4670 2024-03-25T18:22:33.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 >> v1:172.21.15.7:6790/0 conn(0x7f486013c480 legacy=0x7f486014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 >> v1:172.21.15.183:6789/0 conn(0x7f48600b4ff0 legacy=0x7f486013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:22:33.848 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 1 monclient: found mon.noname-a 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48600a9b70 con 0x7f48600b4670 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 10 monclient: _finish_auth 0 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f484c001690 con 0x7f48600b4670 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f485f7fe640 10 monclient: dump: 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.849 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f4866b3e640 5 monclient: authenticate success, global_id 4569 2024-03-25T18:22:33.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.846+0000 7f4866b3e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f484c0017f0 con 0x7f48600b4670 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f485f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f485f7fe640 1 -- 172.21.15.7:0/3351749666 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f484c001c00 con 0x7f48600b4670 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f485f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f485f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f485f7fe640 10 monclient: dump: 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:33.881 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:33.882 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:33.882 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:33.882 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:33.882 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:33.882 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f4866b3e640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:33.882 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.879+0000 7f4866b3e640 4 set_mon_vals no callback set 2024-03-25T18:22:33.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.880+0000 7f4866b3e640 10 monclient: shutdown 2024-03-25T18:22:33.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.880+0000 7f4866b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:33.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.880+0000 7f4866b3e640 1 -- 172.21.15.7:0/3351749666 >> v1:172.21.15.7:6789/0 conn(0x7f48600b4670 legacy=0x7f48601425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.884 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.880+0000 7f4866b3e640 1 -- 172.21.15.7:0/3351749666 shutdown_connections 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.880+0000 7f4866b3e640 1 -- 172.21.15.7:0/3351749666 >> 172.21.15.7:0/3351749666 conn(0x7f48600617a0 msgr2=0x7f4860061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.880+0000 7f4866b3e640 1 -- 172.21.15.7:0/3351749666 shutdown_connections 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.880+0000 7f4866b3e640 1 -- 172.21.15.7:0/3351749666 wait complete. 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 10 monclient: build_initial_monmap 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 10 monclient: monmap: 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:33.885 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 1 librados: starting msgr at 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 1 librados: starting objecter 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 1 Processor -- start 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 1 -- start start 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 1 librados: setting wanted keys 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 1 librados: calling monclient init 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.881+0000 7f4866b3e640 10 monclient: init 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.882+0000 7f4866b3e640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.882+0000 7f4866b3e640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:22:33.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.882+0000 7f4866b3e640 10 monclient(hunting): picked mon.noname-c con 0x7f48600b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:33.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.882+0000 7f4866b3e640 10 monclient(hunting): picked mon.noname-b con 0x7f48600b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:33.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.882+0000 7f4866b3e640 10 monclient(hunting): picked mon.noname-a con 0x7f486013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:33.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.882+0000 7f4866b3e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4860144740 con 0x7f486013c480 2024-03-25T18:22:33.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.882+0000 7f4866b3e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4860130b60 con 0x7f48600b4ff0 2024-03-25T18:22:33.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.883+0000 7f4866b3e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f486013c930 con 0x7f48600b4670 2024-03-25T18:22:33.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.883+0000 7f4866b3e640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:33.887 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.883+0000 7f4866b3e640 10 monclient(hunting): authenticate will time out at 7795.987663s 2024-03-25T18:22:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.897+0000 7f485ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f48600b4ff0 0x7f486012fd40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:45888/0 (socket says 172.21.15.7:45888) 2024-03-25T18:22:33.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.897+0000 7f485ffff640 1 -- 172.21.15.7:0/3734413477 learned_addr learned my addr 172.21.15.7:0/3734413477 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:33.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.900+0000 7f485d7fa640 10 client.?.objecter ms_handle_connect 0x7f486013c480 2024-03-25T18:22:33.901 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.900+0000 7f485d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:33.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.902+0000 7f485d7fa640 10 client.?.objecter ms_handle_connect 0x7f48600b4ff0 2024-03-25T18:22:33.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.902+0000 7f485d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:33.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.904+0000 7f485d7fa640 10 client.?.objecter ms_handle_connect 0x7f48600b4670 2024-03-25T18:22:33.905 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.904+0000 7f485d7fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:33.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.912+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1673399948 0 0) 0x7f4860130b60 con 0x7f48600b4ff0 2024-03-25T18:22:33.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.912+0000 7f485d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.912+0000 7f485d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.912+0000 7f485d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.912+0000 7f485d7fa640 10 monclient(hunting): my global_id is 4573 2024-03-25T18:22:33.914 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.912+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4844004eb0 con 0x7f48600b4ff0 2024-03-25T18:22:33.922 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.921+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1696226161 0 0) 0x7f486013c930 con 0x7f48600b4670 2024-03-25T18:22:33.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.921+0000 7f485d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.921+0000 7f485d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.921+0000 7f485d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.921+0000 7f485d7fa640 10 monclient(hunting): my global_id is 4604 2024-03-25T18:22:33.923 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.921+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4860130b60 con 0x7f48600b4670 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 56948805 0 0) 0x7f4844004eb0 con 0x7f48600b4ff0 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f486013c930 con 0x7f48600b4ff0 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f484c001bc0 con 0x7f48600b4ff0 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): dump: 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:33.939 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:33.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 313268006 0 0) 0x7f4860144740 con 0x7f486013c480 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): _negotiate 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 10 monclient(hunting): my global_id is 4572 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4844004eb0 con 0x7f486013c480 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 191278363 0 0) 0x7f4860130b60 con 0x7f48600b4670 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.938+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4860144740 con 0x7f48600b4670 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.939+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4854001970 con 0x7f48600b4670 2024-03-25T18:22:33.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.939+0000 7f485d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.939+0000 7f485d7fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.939+0000 7f485d7fa640 10 monclient(hunting): dump: 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:33.942 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:33.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1291311303 0 0) 0x7f4860144740 con 0x7f48600b4670 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 >> v1:172.21.15.183:6789/0 conn(0x7f48600b4ff0 legacy=0x7f486012fd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 >> v1:172.21.15.7:6789/0 conn(0x7f486013c480 legacy=0x7f4860130450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 10 monclient: _finish_hunting 0 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 1 monclient: found mon.c 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4860144970 con 0x7f48600b4670 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 10 monclient: _finish_auth 0 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 10 monclient: _check_auth_tickets 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f485d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 5 monclient: authenticate success, global_id 4604 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 10 monclient: _renew_subs 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4860144c50 con 0x7f48600b4670 2024-03-25T18:22:33.952 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:33.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 10 monclient: _renew_subs 2024-03-25T18:22:33.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:33.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f486013f280 con 0x7f48600b4670 2024-03-25T18:22:33.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.950+0000 7f4866b3e640 1 librados: init done 2024-03-25T18:22:33.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.951+0000 7f4866b3e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:33.953 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.951+0000 7f4866b3e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:33.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.951+0000 7f4866b3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:33.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:33.951+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f486006dbd0 con 0x7f48600b4670 2024-03-25T18:22:34.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.257+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4854003610 con 0x7f48600b4670 2024-03-25T18:22:34.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.257+0000 7f485d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:34.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.257+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4854003920 con 0x7f48600b4670 2024-03-25T18:22:34.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.257+0000 7f485d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:34.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.257+0000 7f485d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.257+0000 7f485d7fa640 10 monclient: dump: 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:34.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.257+0000 7f485effd640 4 set_mon_vals no callback set 2024-03-25T18:22:34.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.258+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f48540198f0 con 0x7f48600b4670 2024-03-25T18:22:34.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.258+0000 7f485ffff640 10 monclient: get_auth_request con 0x7f484406df50 auth_method 0 2024-03-25T18:22:34.260 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.258+0000 7f485d7fa640 10 client.4604.objecter ms_handle_connect 0x7f484406df50 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f48540535d0 con 0x7f48600b4670 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 10 client.4604.objecter ms_dispatch 0x7f4860000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f485404ef10 con 0x7f48600b4670 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:34.265 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.263+0000 7f485d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:34.314 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:40.951 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.368 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:41.004 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.411 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:08:41.048 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.437 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:41.074 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.438 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:41.064 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.443 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:41.069 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.449 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:41.086 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.473+0000 7f4866b3e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:34.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.473+0000 7f4866b3e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:34.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.473+0000 7f4866b3e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:34.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.473+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48600688c0 con 0x7f48600b4670 2024-03-25T18:22:34.477 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:08:41.103 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.495+0000 7f485d7fa640 1 -- 172.21.15.7:0/3734413477 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f4854051010 con 0x7f48600b4670 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.495+0000 7f485d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.495+0000 7f485d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:34.496 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:34.497 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:34.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.497+0000 7f4866b3e640 10 librados: watch_flush enter 2024-03-25T18:22:34.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.497+0000 7f4866b3e640 10 librados: watch_flush exit 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.497+0000 7f4866b3e640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.497+0000 7f4866b3e640 10 client.4604.objecter successfully canceled tick 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 >> v1:172.21.15.7:6800/1717564767 conn(0x7f484406df50 legacy=0x7f4844070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 10 monclient: shutdown 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 >> v1:172.21.15.7:6790/0 conn(0x7f48600b4670 legacy=0x7f4860145d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 shutdown_connections 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 >> 172.21.15.7:0/3734413477 conn(0x7f48600617a0 msgr2=0x7f48601407f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 shutdown_connections 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 1 -- 172.21.15.7:0/3734413477 wait complete. 2024-03-25T18:22:34.500 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:34.498+0000 7f4866b3e640 1 librados: shutdown 2024-03-25T18:22:34.506 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:08:41.143 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:34.510 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:41.316 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:41.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.430+0000 7f541a474640 10 monclient: get_monmap_and_config 2024-03-25T18:22:41.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.430+0000 7f541a474640 10 monclient: build_initial_monmap 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.430+0000 7f541a474640 10 monclient: monmap: 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:41.432846+0000 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:41.432846+0000 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:41.433 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.431+0000 7f541a474640 1 Processor -- start 2024-03-25T18:22:41.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.431+0000 7f541a474640 1 -- start start 2024-03-25T18:22:41.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.431+0000 7f541a474640 10 monclient: init 2024-03-25T18:22:41.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:41.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:41.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 10 monclient(hunting): picked mon.noname-a con 0x7f54140b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:41.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 10 monclient(hunting): picked mon.noname-c con 0x7f54140b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:41.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 10 monclient(hunting): picked mon.noname-b con 0x7f541413c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:41.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140ab0e0 con 0x7f54140b4ff0 2024-03-25T18:22:41.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140ab3c0 con 0x7f54140b4670 2024-03-25T18:22:41.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54140a9f30 con 0x7f541413c480 2024-03-25T18:22:41.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:41.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.432+0000 7f541a474640 10 monclient(hunting): authenticate will time out at 7803.538046s 2024-03-25T18:22:41.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.440+0000 7f5418a11640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f541413c480 0x7f541414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:55094/0 (socket says 172.21.15.7:55094) 2024-03-25T18:22:41.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.440+0000 7f5418a11640 1 -- 172.21.15.7:0/3852250230 learned_addr learned my addr 172.21.15.7:0/3852250230 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:41.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.451+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897165071 0 0) 0x7f54140a9f30 con 0x7f541413c480 2024-03-25T18:22:41.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.451+0000 7f5412ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.451+0000 7f5412ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.451+0000 7f5412ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.451+0000 7f5412ffd640 10 monclient(hunting): my global_id is 4607 2024-03-25T18:22:41.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.451+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f53fc003520 con 0x7f541413c480 2024-03-25T18:22:41.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4014125109 0 0) 0x7f53fc003520 con 0x7f541413c480 2024-03-25T18:22:41.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 >> v1:172.21.15.183:6789/0 conn(0x7f54140b4670 legacy=0x7f54141425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 >> v1:172.21.15.7:6789/0 conn(0x7f54140b4ff0 legacy=0x7f541413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:22:41.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:41.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 1 monclient: found mon.noname-b 2024-03-25T18:22:41.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:41.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:22:41.462 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54140a9b70 con 0x7f541413c480 2024-03-25T18:22:41.462 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:41.462 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:41.462 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f5412ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:41.462 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f541a474640 5 monclient: authenticate success, global_id 4607 2024-03-25T18:22:41.462 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.459+0000 7f541a474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.464+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f540c001a10 con 0x7f541413c480 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.464+0000 7f5412ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.464+0000 7f5412ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.464+0000 7f5412ffd640 10 monclient: dump: 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.466 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.467 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:41.467 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:41.467 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:41.467 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:41.467 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.464+0000 7f541a474640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f540c001ce0 con 0x7f541413c480 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f5412ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f5412ffd640 1 -- 172.21.15.7:0/3852250230 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f540c002070 con 0x7f541413c480 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f541a474640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f541a474640 4 set_mon_vals no callback set 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f5412ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f5412ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.470+0000 7f5412ffd640 10 monclient: dump: 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:41.472 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:41.473 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:41.473 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:41.473 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.473 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.471+0000 7f541a474640 10 monclient: shutdown 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.471+0000 7f541a474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.471+0000 7f541a474640 1 -- 172.21.15.7:0/3852250230 >> v1:172.21.15.7:6790/0 conn(0x7f541413c480 legacy=0x7f541414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.471+0000 7f541a474640 1 -- 172.21.15.7:0/3852250230 shutdown_connections 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.471+0000 7f541a474640 1 -- 172.21.15.7:0/3852250230 >> 172.21.15.7:0/3852250230 conn(0x7f54140617a0 msgr2=0x7f5414061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.471+0000 7f541a474640 1 -- 172.21.15.7:0/3852250230 shutdown_connections 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.471+0000 7f541a474640 1 -- 172.21.15.7:0/3852250230 wait complete. 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 10 monclient: build_initial_monmap 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 10 monclient: monmap: 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:41.474 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:41.475 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:41.475 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:41.475 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 1 librados: starting msgr at 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 1 librados: starting objecter 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 1 Processor -- start 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 1 -- start start 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 1 librados: setting wanted keys 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 1 librados: calling monclient init 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.472+0000 7f541a474640 10 monclient: init 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 10 monclient(hunting): picked mon.noname-a con 0x7f54140b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 10 monclient(hunting): picked mon.noname-c con 0x7f54140b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:41.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 10 monclient(hunting): picked mon.noname-b con 0x7f541413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:41.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5414130d50 con 0x7f54140b4670 2024-03-25T18:22:41.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5414144670 con 0x7f541413c480 2024-03-25T18:22:41.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54141448e0 con 0x7f54140b4ff0 2024-03-25T18:22:41.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:41.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.473+0000 7f541a474640 10 monclient(hunting): authenticate will time out at 7803.579158s 2024-03-25T18:22:41.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.478+0000 7f5418a11640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f541413c480 0x7f54141e28f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:55674/0 (socket says 172.21.15.7:55674) 2024-03-25T18:22:41.480 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.478+0000 7f5418a11640 1 -- 172.21.15.7:0/1939998753 learned_addr learned my addr 172.21.15.7:0/1939998753 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:41.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.483+0000 7f5410ff9640 10 client.?.objecter ms_handle_connect 0x7f541413c480 2024-03-25T18:22:41.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.483+0000 7f5410ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:41.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.495+0000 7f5410ff9640 10 client.?.objecter ms_handle_connect 0x7f54140b4ff0 2024-03-25T18:22:41.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.495+0000 7f5410ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:41.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.495+0000 7f5410ff9640 10 client.?.objecter ms_handle_connect 0x7f54140b4670 2024-03-25T18:22:41.497 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.495+0000 7f5410ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:41.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.522+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669861461 0 0) 0x7f5404001090 con 0x7f54140b4670 2024-03-25T18:22:41.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.522+0000 7f5410ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.522+0000 7f5410ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.522+0000 7f5410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.522+0000 7f5410ff9640 10 monclient(hunting): my global_id is 4575 2024-03-25T18:22:41.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.522+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5400004eb0 con 0x7f54140b4670 2024-03-25T18:22:41.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.582+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4292564453 0 0) 0x7f5414144670 con 0x7f541413c480 2024-03-25T18:22:41.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.582+0000 7f5410ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.582+0000 7f5410ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.582+0000 7f5410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.582+0000 7f5410ff9640 10 monclient(hunting): my global_id is 4579 2024-03-25T18:22:41.584 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.582+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5404001090 con 0x7f541413c480 2024-03-25T18:22:41.587 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.585+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1116714973 0 0) 0x7f54141448e0 con 0x7f54140b4ff0 2024-03-25T18:22:41.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.585+0000 7f5410ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:22:41.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.585+0000 7f5410ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:41.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.585+0000 7f5410ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:41.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.585+0000 7f5410ff9640 10 monclient(hunting): my global_id is 4610 2024-03-25T18:22:41.588 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.585+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5414144670 con 0x7f54140b4ff0 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.605+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4085261793 0 0) 0x7f5414144670 con 0x7f54140b4ff0 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.605+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f54141448e0 con 0x7f54140b4ff0 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.605+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5408001860 con 0x7f54140b4ff0 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.605+0000 7f5410ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.605+0000 7f5410ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:41.607 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.605+0000 7f5410ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.608 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.605+0000 7f5410ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.610+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3705297739 0 0) 0x7f5404001090 con 0x7f541413c480 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.610+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5414144670 con 0x7f541413c480 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.610+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f540c0019d0 con 0x7f541413c480 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.610+0000 7f5410ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.610+0000 7f5410ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.610+0000 7f5410ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:41.613 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:41.614 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.618+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4175051648 0 0) 0x7f5400004eb0 con 0x7f54140b4670 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.618+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5404001090 con 0x7f54140b4670 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.618+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5404001860 con 0x7f54140b4670 2024-03-25T18:22:41.619 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.618+0000 7f5410ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.618+0000 7f5410ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.618+0000 7f5410ff9640 10 monclient(hunting): dump: 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:41.620 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3869581558 0 0) 0x7f5414144670 con 0x7f541413c480 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 >> v1:172.21.15.7:6790/0 conn(0x7f54140b4ff0 legacy=0x7f54141dbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 >> v1:172.21.15.7:6789/0 conn(0x7f54140b4670 legacy=0x7f54141db440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 1 monclient: found mon.b 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:41.624 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5414144bc0 con 0x7f541413c480 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 10 monclient: _finish_auth 0 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f5410ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f541a474640 5 monclient: authenticate success, global_id 4579 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f541a474640 10 monclient: _renew_subs 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f541a474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.621+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5414144f00 con 0x7f541413c480 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.622+0000 7f541a474640 10 client.4579.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.622+0000 7f541a474640 10 monclient: _renew_subs 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.622+0000 7f541a474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.622+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f541413f470 con 0x7f541413c480 2024-03-25T18:22:41.625 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.622+0000 7f541a474640 1 librados: init done 2024-03-25T18:22:41.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.623+0000 7f541a474640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:41.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.623+0000 7f541a474640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:41.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.623+0000 7f541a474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:41.626 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.623+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f541406dbd0 con 0x7f541413c480 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.935+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f540c004e10 con 0x7f541413c480 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.935+0000 7f5410ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.935+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f540c005120 con 0x7f541413c480 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.935+0000 7f5410ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.935+0000 7f5410ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.935+0000 7f5410ff9640 10 monclient: dump: 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:41.938 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:41.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.935+0000 7f54127fc640 4 set_mon_vals no callback set 2024-03-25T18:22:41.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.936+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f540c01b0f0 con 0x7f541413c480 2024-03-25T18:22:41.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.937+0000 7f5413fff640 10 monclient: get_auth_request con 0x7f540006e110 auth_method 0 2024-03-25T18:22:41.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.937+0000 7f5410ff9640 10 client.4579.objecter ms_handle_connect 0x7f540006e110 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f540c054c20 con 0x7f541413c480 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 10 client.4579.objecter ms_dispatch 0x7f5414000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 3 client.4579.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 3 client.4579.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 20 client.4579.objecter dump_active .. 0 homeless 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f540c050730 con 0x7f541413c480 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:41.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:41.941+0000 7f5410ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:42.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.197+0000 7f541a474640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.197+0000 7f541a474640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.198+0000 7f541a474640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:42.199 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.198+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54140b19b0 con 0x7f541413c480 2024-03-25T18:22:42.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.232+0000 7f5410ff9640 1 -- 172.21.15.7:0/1939998753 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f540c052660 con 0x7f541413c480 2024-03-25T18:22:42.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.232+0000 7f5410ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:42.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.232+0000 7f5410ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:42.234 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:42.234 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 10 librados: watch_flush enter 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 10 librados: watch_flush exit 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 20 client.4579.objecter shutdown clearing up homeless session... 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 10 client.4579.objecter successfully canceled tick 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 >> v1:172.21.15.7:6800/1717564767 conn(0x7f540006e110 legacy=0x7f54000705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 10 monclient: shutdown 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 >> v1:172.21.15.183:6789/0 conn(0x7f541413c480 legacy=0x7f54141e28f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 shutdown_connections 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 >> 172.21.15.7:0/1939998753 conn(0x7f54140617a0 msgr2=0x7f54140b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.235+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 shutdown_connections 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.236+0000 7f541a474640 1 -- 172.21.15.7:0/1939998753 wait complete. 2024-03-25T18:22:42.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:42.236+0000 7f541a474640 1 librados: shutdown 2024-03-25T18:22:42.248 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:49.055 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:49.170 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.169+0000 7f1f88da8640 10 monclient: get_monmap_and_config 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.169+0000 7f1f88da8640 10 monclient: build_initial_monmap 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.169+0000 7f1f88da8640 10 monclient: monmap: 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:49.170671+0000 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:49.170671+0000 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:49.171 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.170+0000 7f1f88da8640 1 Processor -- start 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.170+0000 7f1f88da8640 1 -- start start 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.170+0000 7f1f88da8640 10 monclient: init 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8406c3c0 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 10 monclient(hunting): picked mon.noname-c con 0x7f1f840b0f00 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 10 monclient(hunting): picked mon.noname-b con 0x7f1f840aee80 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:49.172 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840b1390 con 0x7f1f8406c3c0 2024-03-25T18:22:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f8406dbe0 con 0x7f1f840b0f00 2024-03-25T18:22:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f840adab0 con 0x7f1f840aee80 2024-03-25T18:22:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:49.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.171+0000 7f1f88da8640 10 monclient(hunting): authenticate will time out at 7811.275940s 2024-03-25T18:22:49.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.191+0000 7f1f81d9b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1f840b0f00 0x7f1f840ae770 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:60340/0 (socket says 172.21.15.7:60340) 2024-03-25T18:22:49.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.191+0000 7f1f81d9b640 1 -- 172.21.15.7:0/147190267 learned_addr learned my addr 172.21.15.7:0/147190267 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:49.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.203+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3336451996 0 0) 0x7f1f840b1390 con 0x7f1f8406c3c0 2024-03-25T18:22:49.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.203+0000 7f1f8159a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.203+0000 7f1f8159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.203+0000 7f1f8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.203+0000 7f1f8159a640 10 monclient(hunting): my global_id is 4578 2024-03-25T18:22:49.204 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.203+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f6c003520 con 0x7f1f8406c3c0 2024-03-25T18:22:49.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.208+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3531933130 0 0) 0x7f1f8406dbe0 con 0x7f1f840b0f00 2024-03-25T18:22:49.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.208+0000 7f1f8159a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.208+0000 7f1f8159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.208+0000 7f1f8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.209 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.208+0000 7f1f8159a640 10 monclient(hunting): my global_id is 4582 2024-03-25T18:22:49.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.208+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f840b1390 con 0x7f1f840b0f00 2024-03-25T18:22:49.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.209+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1655304492 0 0) 0x7f1f840adab0 con 0x7f1f840aee80 2024-03-25T18:22:49.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.209+0000 7f1f8159a640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.209+0000 7f1f8159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.209+0000 7f1f8159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.209+0000 7f1f8159a640 10 monclient(hunting): my global_id is 4613 2024-03-25T18:22:49.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.209+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f8406dbe0 con 0x7f1f840aee80 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2885550856 0 0) 0x7f1f6c003520 con 0x7f1f8406c3c0 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 >> v1:172.21.15.7:6790/0 conn(0x7f1f840aee80 legacy=0x7f1f8414df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 >> v1:172.21.15.183:6789/0 conn(0x7f1f840b0f00 legacy=0x7f1f840ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 10 monclient: _finish_hunting 0 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 1 monclient: found mon.noname-a 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f840add90 con 0x7f1f8406c3c0 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 10 monclient: _finish_auth 0 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 10 monclient: _check_auth_tickets 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1f74001690 con 0x7f1f8406c3c0 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f88da8640 5 monclient: authenticate success, global_id 4578 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f88da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.217 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f8159a640 10 monclient: dump: 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:49.218 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.215+0000 7f1f88da8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f740017f0 con 0x7f1f8406c3c0 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f8159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f8159a640 1 -- 172.21.15.7:0/147190267 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1f74001c00 con 0x7f1f8406c3c0 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f8159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f8159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f8159a640 10 monclient: dump: 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:49.233 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f88da8640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:49.234 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.232+0000 7f1f88da8640 4 set_mon_vals no callback set 2024-03-25T18:22:49.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.233+0000 7f1f88da8640 10 monclient: shutdown 2024-03-25T18:22:49.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.233+0000 7f1f88da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:49.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.233+0000 7f1f88da8640 1 -- 172.21.15.7:0/147190267 >> v1:172.21.15.7:6789/0 conn(0x7f1f8406c3c0 legacy=0x7f1f840b07f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.233+0000 7f1f88da8640 1 -- 172.21.15.7:0/147190267 shutdown_connections 2024-03-25T18:22:49.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.233+0000 7f1f88da8640 1 -- 172.21.15.7:0/147190267 >> 172.21.15.7:0/147190267 conn(0x7f1f840b5df0 msgr2=0x7f1f840b6200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:49.236 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.233+0000 7f1f88da8640 1 -- 172.21.15.7:0/147190267 shutdown_connections 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.233+0000 7f1f88da8640 1 -- 172.21.15.7:0/147190267 wait complete. 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 10 monclient: build_initial_monmap 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 10 monclient: monmap: 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:49.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 1 librados: starting msgr at 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 1 librados: starting objecter 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 1 Processor -- start 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 1 -- start start 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 1 librados: setting wanted keys 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 1 librados: calling monclient init 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.234+0000 7f1f88da8640 10 monclient: init 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 10 monclient(hunting): picked mon.noname-a con 0x7f1f8406c3c0 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 10 monclient(hunting): picked mon.noname-b con 0x7f1f840aee80 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 10 monclient(hunting): picked mon.noname-c con 0x7f1f840b0f00 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f841a2d00 con 0x7f1f8406c3c0 2024-03-25T18:22:49.238 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f841c3260 con 0x7f1f840aee80 2024-03-25T18:22:49.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1f841a3a60 con 0x7f1f840b0f00 2024-03-25T18:22:49.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:49.239 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.235+0000 7f1f88da8640 10 monclient(hunting): authenticate will time out at 7811.340020s 2024-03-25T18:22:49.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.245+0000 7f1f81d9b640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f1f840aee80 0x7f1f841c2440 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:60388/0 (socket says 172.21.15.7:60388) 2024-03-25T18:22:49.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.245+0000 7f1f81d9b640 1 -- 172.21.15.7:0/3311491646 learned_addr learned my addr 172.21.15.7:0/3311491646 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:49.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.246+0000 7f1f72ffd640 10 client.?.objecter ms_handle_connect 0x7f1f840aee80 2024-03-25T18:22:49.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.246+0000 7f1f72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.249+0000 7f1f72ffd640 10 client.?.objecter ms_handle_connect 0x7f1f840b0f00 2024-03-25T18:22:49.250 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.249+0000 7f1f72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.256+0000 7f1f72ffd640 10 client.?.objecter ms_handle_connect 0x7f1f8406c3c0 2024-03-25T18:22:49.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.256+0000 7f1f72ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:49.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.261+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3050093624 0 0) 0x7f1f841a3a60 con 0x7f1f840b0f00 2024-03-25T18:22:49.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.261+0000 7f1f72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.261+0000 7f1f72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.261+0000 7f1f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.261+0000 7f1f72ffd640 10 monclient(hunting): my global_id is 4616 2024-03-25T18:22:49.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.261+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f64004eb0 con 0x7f1f840b0f00 2024-03-25T18:22:49.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.262+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1414555746 0 0) 0x7f1f841c3260 con 0x7f1f840aee80 2024-03-25T18:22:49.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.262+0000 7f1f72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.263 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.262+0000 7f1f72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.262+0000 7f1f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.262+0000 7f1f72ffd640 10 monclient(hunting): my global_id is 4585 2024-03-25T18:22:49.264 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.262+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f841a3a60 con 0x7f1f840aee80 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.278+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2631112465 0 0) 0x7f1f841a3a60 con 0x7f1f840aee80 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.278+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f841c3260 con 0x7f1f840aee80 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.278+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1f780018d0 con 0x7f1f840aee80 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.278+0000 7f1f72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.278+0000 7f1f72ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.278+0000 7f1f72ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.279 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:49.280 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:49.280 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:49.280 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:49.280 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:49.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.278+0000 7f1f72ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.283+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3350885429 0 0) 0x7f1f64004eb0 con 0x7f1f840b0f00 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.283+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1f841a3a60 con 0x7f1f840b0f00 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.283+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1f7c001c70 con 0x7f1f840b0f00 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.283+0000 7f1f72ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.283+0000 7f1f72ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.283+0000 7f1f72ffd640 10 monclient(hunting): dump: 2024-03-25T18:22:49.284 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:49.285 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:49.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.285+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1766586971 0 0) 0x7f1f841a2d00 con 0x7f1f8406c3c0 2024-03-25T18:22:49.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.285+0000 7f1f72ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:22:49.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.285+0000 7f1f72ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:49.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.285+0000 7f1f72ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:49.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.285+0000 7f1f72ffd640 10 monclient(hunting): my global_id is 4581 2024-03-25T18:22:49.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.285+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1f64004eb0 con 0x7f1f8406c3c0 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 917654327 0 0) 0x7f1f841c3260 con 0x7f1f840aee80 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 >> v1:172.21.15.7:6790/0 conn(0x7f1f840b0f00 legacy=0x7f1f841c2b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 >> v1:172.21.15.7:6789/0 conn(0x7f1f8406c3c0 legacy=0x7f1f841da8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 1 monclient: found mon.b 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f841c39d0 con 0x7f1f840aee80 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 10 monclient: _finish_auth 0 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f72ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:49.293 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 5 monclient: authenticate success, global_id 4585 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 10 monclient: _renew_subs 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f841c65c0 con 0x7f1f840aee80 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 10 client.4585.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 10 monclient: _renew_subs 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f841db190 con 0x7f1f840aee80 2024-03-25T18:22:49.294 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.291+0000 7f1f88da8640 1 librados: init done 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.292+0000 7f1f88da8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.292+0000 7f1f88da8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.292+0000 7f1f88da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:49.295 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.292+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f840b2630 con 0x7f1f840aee80 2024-03-25T18:22:49.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.311+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1f780047e0 con 0x7f1f840aee80 2024-03-25T18:22:49.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.311+0000 7f1f72ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:49.312 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.311+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1f78004af0 con 0x7f1f840aee80 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.311+0000 7f1f72ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.311+0000 7f1f72ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.311+0000 7f1f72ffd640 10 monclient: dump: 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:49.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.311+0000 7f1f80d99640 4 set_mon_vals no callback set 2024-03-25T18:22:49.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.312+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f1f7801aac0 con 0x7f1f840aee80 2024-03-25T18:22:49.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.313+0000 7f1f8259c640 10 monclient: get_auth_request con 0x7f1f6406df50 auth_method 0 2024-03-25T18:22:49.314 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.314+0000 7f1f72ffd640 10 client.4585.objecter ms_handle_connect 0x7f1f6406df50 2024-03-25T18:22:49.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.314+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f1f78054740 con 0x7f1f840aee80 2024-03-25T18:22:49.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.314+0000 7f1f72ffd640 10 client.4585.objecter ms_dispatch 0x7f1f84000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:49.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.314+0000 7f1f72ffd640 3 client.4585.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:49.315 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.314+0000 7f1f72ffd640 3 client.4585.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:49.316 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.314+0000 7f1f72ffd640 20 client.4585.objecter dump_active .. 0 homeless 2024-03-25T18:22:49.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.571+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1f78050250 con 0x7f1f840aee80 2024-03-25T18:22:49.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.571+0000 7f1f72ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:49.572 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.571+0000 7f1f72ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:49.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.796+0000 7f1f88da8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:49.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.796+0000 7f1f88da8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:49.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.796+0000 7f1f88da8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:22:49.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.796+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1f8404f730 con 0x7f1f840aee80 2024-03-25T18:22:49.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.813+0000 7f1f72ffd640 1 -- 172.21.15.7:0/3311491646 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f1f78052180 con 0x7f1f840aee80 2024-03-25T18:22:49.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.813+0000 7f1f72ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:49.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.813+0000 7f1f72ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:49.815 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:49.815 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:49.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.815+0000 7f1f88da8640 10 librados: watch_flush enter 2024-03-25T18:22:49.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 10 librados: watch_flush exit 2024-03-25T18:22:49.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 20 client.4585.objecter shutdown clearing up homeless session... 2024-03-25T18:22:49.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 10 client.4585.objecter successfully canceled tick 2024-03-25T18:22:49.817 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 >> v1:172.21.15.7:6800/1717564767 conn(0x7f1f6406df50 legacy=0x7f1f64070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 10 monclient: shutdown 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 >> v1:172.21.15.183:6789/0 conn(0x7f1f840aee80 legacy=0x7f1f841c2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 shutdown_connections 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 >> 172.21.15.7:0/3311491646 conn(0x7f1f840b5df0 msgr2=0x7f1f841cd2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 shutdown_connections 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 1 -- 172.21.15.7:0/3311491646 wait complete. 2024-03-25T18:22:49.818 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:49.816+0000 7f1f88da8640 1 librados: shutdown 2024-03-25T18:22:49.828 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:22:54.114 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:00.750 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.124 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:00.761 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.163 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:00.799 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.172 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:00.799 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.174 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:00.811 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.177 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:00.803 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.187 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:00.824 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.212 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:00.838 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:54.226 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:00.863 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:22:56.634 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.746+0000 7fb6df8a1640 10 monclient: get_monmap_and_config 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.746+0000 7fb6df8a1640 10 monclient: build_initial_monmap 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.746+0000 7fb6df8a1640 10 monclient: monmap: 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:22:56.747068+0000 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:22:56.747068+0000 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:56.747 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:22:56.748 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:22:56.748 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:56.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.746+0000 7fb6df8a1640 1 Processor -- start 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 1 -- start start 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient: init 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d80b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d80b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80ab0e0 con 0x7fb6d813c480 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80ab3c0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80a9f30 con 0x7fb6d80b4670 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:56.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.747+0000 7fb6df8a1640 10 monclient(hunting): authenticate will time out at 7818.852590s 2024-03-25T18:22:56.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.766+0000 7fb6dde3e640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fb6d813c480 0x7fb6d814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:48340/0 (socket says 172.21.15.7:48340) 2024-03-25T18:22:56.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.766+0000 7fb6dde3e640 1 -- 172.21.15.7:0/2038924754 learned_addr learned my addr 172.21.15.7:0/2038924754 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2147892971 0 0) 0x7fb6d80ab3c0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): my global_id is 4588 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6c4003520 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2290178374 0 0) 0x7fb6d80a9f30 con 0x7fb6d80b4670 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 10 monclient(hunting): my global_id is 4619 2024-03-25T18:22:56.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.777+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d80ab3c0 con 0x7fb6d80b4670 2024-03-25T18:22:56.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3270076320 0 0) 0x7fb6c4003520 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.786 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 >> v1:172.21.15.7:6790/0 conn(0x7fb6d80b4670 legacy=0x7fb6d81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 >> v1:172.21.15.7:6789/0 conn(0x7fb6d813c480 legacy=0x7fb6d814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 10 monclient: _finish_hunting 0 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 1 monclient: found mon.noname-c 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80a9b70 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 10 monclient: _finish_auth 0 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 10 monclient: _check_auth_tickets 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6d4001690 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6cffff640 10 monclient: dump: 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:56.787 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6df8a1640 5 monclient: authenticate success, global_id 4588 2024-03-25T18:22:56.788 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.785+0000 7fb6df8a1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.794+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6d40017f0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.794+0000 7fb6cffff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.794+0000 7fb6cffff640 1 -- 172.21.15.7:0/2038924754 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6d4001c00 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.794+0000 7fb6cffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.794+0000 7fb6cffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.794+0000 7fb6cffff640 10 monclient: dump: 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.796 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.795+0000 7fb6df8a1640 10 monclient: get_monmap_and_config success 2024-03-25T18:22:56.797 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.795+0000 7fb6df8a1640 4 set_mon_vals no callback set 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.795+0000 7fb6df8a1640 10 monclient: shutdown 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.795+0000 7fb6df8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.795+0000 7fb6df8a1640 1 -- 172.21.15.7:0/2038924754 >> v1:172.21.15.183:6789/0 conn(0x7fb6d80b4ff0 legacy=0x7fb6d813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.795+0000 7fb6df8a1640 1 -- 172.21.15.7:0/2038924754 shutdown_connections 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.796+0000 7fb6df8a1640 1 -- 172.21.15.7:0/2038924754 >> 172.21.15.7:0/2038924754 conn(0x7fb6d80617a0 msgr2=0x7fb6d8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.796+0000 7fb6df8a1640 1 -- 172.21.15.7:0/2038924754 shutdown_connections 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.796+0000 7fb6df8a1640 1 -- 172.21.15.7:0/2038924754 wait complete. 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.797+0000 7fb6df8a1640 10 monclient: build_initial_monmap 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.797+0000 7fb6df8a1640 10 monclient: monmap: 2024-03-25T18:22:56.798 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:22:56.799 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.797+0000 7fb6df8a1640 1 librados: starting msgr at 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.798+0000 7fb6df8a1640 1 librados: starting objecter 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.798+0000 7fb6df8a1640 1 Processor -- start 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.798+0000 7fb6df8a1640 1 -- start start 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.798+0000 7fb6df8a1640 1 librados: setting wanted keys 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.798+0000 7fb6df8a1640 1 librados: calling monclient init 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.798+0000 7fb6df8a1640 10 monclient: init 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 10 monclient: _reopen_session rank -1 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 10 monclient(hunting): picked mon.noname-a con 0x7fb6d80b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 10 monclient(hunting): picked mon.noname-c con 0x7fb6d80b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 10 monclient(hunting): picked mon.noname-b con 0x7fb6d813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80b0170 con 0x7fb6d80b4670 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80b03e0 con 0x7fb6d813c480 2024-03-25T18:22:56.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6d80b06c0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 10 monclient(hunting): _renew_subs 2024-03-25T18:22:56.802 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.799+0000 7fb6df8a1640 10 monclient(hunting): authenticate will time out at 7818.904546s 2024-03-25T18:22:56.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.812+0000 7fb6dce3c640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fb6d80b4ff0 0x7fb6d80ae540 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:47928/0 (socket says 172.21.15.7:47928) 2024-03-25T18:22:56.813 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.812+0000 7fb6dce3c640 1 -- 172.21.15.7:0/3876005769 learned_addr learned my addr 172.21.15.7:0/3876005769 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:22:56.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.813+0000 7fb6cdffb640 10 client.?.objecter ms_handle_connect 0x7fb6d80b4ff0 2024-03-25T18:22:56.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.813+0000 7fb6cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:56.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.819+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2578620266 0 0) 0x7fb6d80b06c0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.819+0000 7fb6cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.819+0000 7fb6cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.819+0000 7fb6cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.819+0000 7fb6cdffb640 10 monclient(hunting): my global_id is 4622 2024-03-25T18:22:56.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.819+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6bc004eb0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.821+0000 7fb6cdffb640 10 client.?.objecter ms_handle_connect 0x7fb6d813c480 2024-03-25T18:22:56.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.821+0000 7fb6cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:56.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.825+0000 7fb6cdffb640 10 client.?.objecter ms_handle_connect 0x7fb6d80b4670 2024-03-25T18:22:56.826 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.825+0000 7fb6cdffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:22:56.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.835+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 95882913 0 0) 0x7fb6d80b03e0 con 0x7fb6d813c480 2024-03-25T18:22:56.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.835+0000 7fb6cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.835+0000 7fb6cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.835+0000 7fb6cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.835+0000 7fb6cdffb640 10 monclient(hunting): my global_id is 4591 2024-03-25T18:22:56.837 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.835+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d80b06c0 con 0x7fb6d813c480 2024-03-25T18:22:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.837+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2690207114 0 0) 0x7fb6d80b0170 con 0x7fb6d80b4670 2024-03-25T18:22:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.837+0000 7fb6cdffb640 10 monclient(hunting): _negotiate 2024-03-25T18:22:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.837+0000 7fb6cdffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:22:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.837+0000 7fb6cdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:22:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.837+0000 7fb6cdffb640 10 monclient(hunting): my global_id is 4587 2024-03-25T18:22:56.838 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.837+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb6d80b03e0 con 0x7fb6d80b4670 2024-03-25T18:22:56.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.838+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1863602647 0 0) 0x7fb6bc004eb0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.839 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.838+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6d80b0170 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.838+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6c8001a70 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.838+0000 7fb6cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.838+0000 7fb6cdffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.838+0000 7fb6cdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:56.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.838+0000 7fb6cdffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:22:56.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.845+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2809212170 0 0) 0x7fb6d80b06c0 con 0x7fb6d813c480 2024-03-25T18:22:56.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.845+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6bc004eb0 con 0x7fb6d813c480 2024-03-25T18:22:56.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.845+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6d0001b00 con 0x7fb6d813c480 2024-03-25T18:22:56.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.845+0000 7fb6cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.845+0000 7fb6cdffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:22:56.846 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.845+0000 7fb6cdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:56.847 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.851+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 417558700 0 0) 0x7fb6d80b03e0 con 0x7fb6d80b4670 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.851+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb6d80b06c0 con 0x7fb6d80b4670 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.851+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6d4001b00 con 0x7fb6d80b4670 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.851+0000 7fb6cdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.851+0000 7fb6cdffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.851+0000 7fb6cdffb640 10 monclient(hunting): dump: 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.852 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:56.853 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:56.853 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:56.853 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:56.853 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:56.853 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:56.853 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2282225154 0 0) 0x7fb6d80b0170 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 >> v1:172.21.15.183:6789/0 conn(0x7fb6d813c480 legacy=0x7fb6d81e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 >> v1:172.21.15.7:6789/0 conn(0x7fb6d80b4670 legacy=0x7fb6d80b1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 10 monclient: _finish_hunting 0 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 1 monclient: found mon.c 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d80b09a0 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 10 monclient: _finish_auth 0 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 10 monclient: _check_auth_tickets 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6cdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 5 monclient: authenticate success, global_id 4622 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 10 monclient: _renew_subs 2024-03-25T18:22:56.855 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6d80b0f60 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 10 monclient: _renew_subs 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6d80aec50 con 0x7fb6d80b4ff0 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.853+0000 7fb6df8a1640 1 librados: init done 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.854+0000 7fb6df8a1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.854+0000 7fb6df8a1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.854+0000 7fb6df8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:56.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:56.854+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6d806dbd0 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.172+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb6c8001cd0 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.172+0000 7fb6cdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.172+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fb6c8003a40 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.172+0000 7fb6cdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.172+0000 7fb6cdffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.172+0000 7fb6cdffb640 10 monclient: dump: 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:22:57.173 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:22:57.174 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.172+0000 7fb6cf7fe640 4 set_mon_vals no callback set 2024-03-25T18:22:57.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.173+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fb6c8003c60 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.173+0000 7fb6dd63d640 10 monclient: get_auth_request con 0x7fb6bc06e0b0 auth_method 0 2024-03-25T18:22:57.175 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.174+0000 7fb6cdffb640 10 client.4622.objecter ms_handle_connect 0x7fb6bc06e0b0 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fb6c80548b0 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 10 client.4622.objecter ms_dispatch 0x7fb6d8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb6c80503c0 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:22:57.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.181+0000 7fb6cdffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.430+0000 7fb6df8a1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.430+0000 7fb6df8a1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.430+0000 7fb6df8a1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:22:57.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.430+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb6d80688c0 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.453+0000 7fb6cdffb640 1 -- 172.21.15.7:0/3876005769 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fb6c80522f0 con 0x7fb6d80b4ff0 2024-03-25T18:22:57.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.453+0000 7fb6cdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:22:57.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.453+0000 7fb6cdffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:22:57.454 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:22:57.454 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:22:57.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.455+0000 7fb6df8a1640 10 librados: watch_flush enter 2024-03-25T18:22:57.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 10 librados: watch_flush exit 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 10 client.4622.objecter successfully canceled tick 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 >> v1:172.21.15.7:6800/1717564767 conn(0x7fb6bc06e0b0 legacy=0x7fb6bc070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 10 monclient: shutdown 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 >> v1:172.21.15.7:6790/0 conn(0x7fb6d80b4ff0 legacy=0x7fb6d80ae540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 shutdown_connections 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 >> 172.21.15.7:0/3876005769 conn(0x7fb6d80617a0 msgr2=0x7fb6d80b6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 shutdown_connections 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 1 -- 172.21.15.7:0/3876005769 wait complete. 2024-03-25T18:22:57.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:22:57.456+0000 7fb6df8a1640 1 librados: shutdown 2024-03-25T18:22:57.469 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:04.275 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.392+0000 7f4525406640 10 monclient: get_monmap_and_config 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.392+0000 7f4525406640 10 monclient: build_initial_monmap 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.392+0000 7f4525406640 10 monclient: monmap: 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:04.393068+0000 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:04.393068+0000 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:04.393 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:04.394 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:04.394 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:04.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.392+0000 7f4525406640 1 Processor -- start 2024-03-25T18:23:04.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.393+0000 7f4525406640 1 -- start start 2024-03-25T18:23:04.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.393+0000 7f4525406640 10 monclient: init 2024-03-25T18:23:04.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.393+0000 7f4525406640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:04.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.393+0000 7f4525406640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:04.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.393+0000 7f4525406640 10 monclient(hunting): picked mon.noname-b con 0x7f45200b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:04.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.393+0000 7f4525406640 10 monclient(hunting): picked mon.noname-c con 0x7f45200b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:04.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.394+0000 7f4525406640 10 monclient(hunting): picked mon.noname-a con 0x7f452013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:04.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.394+0000 7f4525406640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45200ab0e0 con 0x7f452013c480 2024-03-25T18:23:04.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.394+0000 7f4525406640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45200ab3c0 con 0x7f45200b4670 2024-03-25T18:23:04.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.394+0000 7f4525406640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45200a9f30 con 0x7f45200b4ff0 2024-03-25T18:23:04.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.394+0000 7f4525406640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:04.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.394+0000 7f4525406640 10 monclient(hunting): authenticate will time out at 7826.498755s 2024-03-25T18:23:04.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.407+0000 7f451effd640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f45200b4ff0 0x7f452013bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:52650/0 (socket says 172.21.15.7:52650) 2024-03-25T18:23:04.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.407+0000 7f451effd640 1 -- 172.21.15.7:0/1501205456 learned_addr learned my addr 172.21.15.7:0/1501205456 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:04.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.425+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1714735861 0 0) 0x7f45200ab3c0 con 0x7f45200b4670 2024-03-25T18:23:04.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.425+0000 7f451dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.425+0000 7f451dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.425+0000 7f451dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.425+0000 7f451dffb640 10 monclient(hunting): my global_id is 4594 2024-03-25T18:23:04.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.425+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4508003520 con 0x7f45200b4670 2024-03-25T18:23:04.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.429+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118425818 0 0) 0x7f45200a9f30 con 0x7f45200b4ff0 2024-03-25T18:23:04.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.429+0000 7f451dffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.429+0000 7f451dffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.429+0000 7f451dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.429+0000 7f451dffb640 10 monclient(hunting): my global_id is 4625 2024-03-25T18:23:04.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.429+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45200ab3c0 con 0x7f45200b4ff0 2024-03-25T18:23:04.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 376036065 0 0) 0x7f4508003520 con 0x7f45200b4670 2024-03-25T18:23:04.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 >> v1:172.21.15.7:6790/0 conn(0x7f45200b4ff0 legacy=0x7f452013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 >> v1:172.21.15.7:6789/0 conn(0x7f452013c480 legacy=0x7f452014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:04.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 1 monclient: found mon.noname-c 2024-03-25T18:23:04.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:04.431 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45200a9b70 con 0x7f45200b4670 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4514001690 con 0x7f45200b4670 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f451dffb640 10 monclient: dump: 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.432 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:04.433 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:04.433 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:04.433 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:04.433 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:04.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f4525406640 5 monclient: authenticate success, global_id 4594 2024-03-25T18:23:04.433 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.430+0000 7f4525406640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:04.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f45140017f0 con 0x7f45200b4670 2024-03-25T18:23:04.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f451dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:04.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f451dffb640 1 -- 172.21.15.7:0/1501205456 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4514001c00 con 0x7f45200b4670 2024-03-25T18:23:04.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f451dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f451dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f451dffb640 10 monclient: dump: 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:04.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f4525406640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:04.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.440+0000 7f4525406640 4 set_mon_vals no callback set 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.441+0000 7f4525406640 10 monclient: shutdown 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.441+0000 7f4525406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.441+0000 7f4525406640 1 -- 172.21.15.7:0/1501205456 >> v1:172.21.15.183:6789/0 conn(0x7f45200b4670 legacy=0x7f45201425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.441+0000 7f4525406640 1 -- 172.21.15.7:0/1501205456 shutdown_connections 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.441+0000 7f4525406640 1 -- 172.21.15.7:0/1501205456 >> 172.21.15.7:0/1501205456 conn(0x7f45200617a0 msgr2=0x7f4520061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.441+0000 7f4525406640 1 -- 172.21.15.7:0/1501205456 shutdown_connections 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.441+0000 7f4525406640 1 -- 172.21.15.7:0/1501205456 wait complete. 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.442+0000 7f4525406640 10 monclient: build_initial_monmap 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.442+0000 7f4525406640 10 monclient: monmap: 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:04.445 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.442+0000 7f4525406640 1 librados: starting msgr at 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.442+0000 7f4525406640 1 librados: starting objecter 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.442+0000 7f4525406640 1 Processor -- start 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 1 -- start start 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 1 librados: setting wanted keys 2024-03-25T18:23:04.446 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 1 librados: calling monclient init 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient: init 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient(hunting): picked mon.noname-a con 0x7f45200b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient(hunting): picked mon.noname-c con 0x7f45200b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient(hunting): picked mon.noname-b con 0x7f452013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4520144660 con 0x7f45200b4670 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4520144940 con 0x7f452013c480 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4520144c20 con 0x7f45200b4ff0 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:04.447 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.443+0000 7f4525406640 10 monclient(hunting): authenticate will time out at 7826.548489s 2024-03-25T18:23:04.456 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.455+0000 7f451e7fc640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f45200b4ff0 0x7f452012fcb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:52674/0 (socket says 172.21.15.7:52674) 2024-03-25T18:23:04.456 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.455+0000 7f451e7fc640 1 -- 172.21.15.7:0/2088609362 learned_addr learned my addr 172.21.15.7:0/2088609362 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:04.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.457+0000 7f45077fe640 10 client.?.objecter ms_handle_connect 0x7f45200b4ff0 2024-03-25T18:23:04.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.457+0000 7f45077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:04.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.458+0000 7f45077fe640 10 client.?.objecter ms_handle_connect 0x7f452013c480 2024-03-25T18:23:04.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.458+0000 7f45077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:04.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.466+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 9144175 0 0) 0x7f4520144940 con 0x7f452013c480 2024-03-25T18:23:04.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.466+0000 7f45077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.466+0000 7f45077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.466+0000 7f45077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.466+0000 7f45077fe640 10 monclient(hunting): my global_id is 4597 2024-03-25T18:23:04.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.466+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c004eb0 con 0x7f452013c480 2024-03-25T18:23:04.470 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.469+0000 7f45077fe640 10 client.?.objecter ms_handle_connect 0x7f45200b4670 2024-03-25T18:23:04.470 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.469+0000 7f45077fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.480+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545758784 0 0) 0x7f450c004eb0 con 0x7f452013c480 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.480+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4520144940 con 0x7f452013c480 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.480+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4518001b20 con 0x7f452013c480 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.480+0000 7f45077fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.480+0000 7f45077fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.480+0000 7f45077fe640 10 monclient(hunting): dump: 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.481 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.480+0000 7f45077fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:04.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.481+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 17538677 0 0) 0x7f4520144c20 con 0x7f45200b4ff0 2024-03-25T18:23:04.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.481+0000 7f45077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.481+0000 7f45077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.481+0000 7f45077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.481+0000 7f45077fe640 10 monclient(hunting): my global_id is 4628 2024-03-25T18:23:04.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.481+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f450c004eb0 con 0x7f45200b4ff0 2024-03-25T18:23:04.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.486+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982526276 0 0) 0x7f4520144660 con 0x7f45200b4670 2024-03-25T18:23:04.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.486+0000 7f45077fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:04.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.486+0000 7f45077fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:04.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.486+0000 7f45077fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:04.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.486+0000 7f45077fe640 10 monclient(hunting): my global_id is 4593 2024-03-25T18:23:04.487 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.486+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4520144c20 con 0x7f45200b4670 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2939158717 0 0) 0x7f4520144940 con 0x7f452013c480 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 >> v1:172.21.15.7:6790/0 conn(0x7f45200b4ff0 legacy=0x7f452012fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 >> v1:172.21.15.7:6789/0 conn(0x7f45200b4670 legacy=0x7f4520145320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 1 monclient: found mon.b 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4520143630 con 0x7f452013c480 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:04.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f45077fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 5 monclient: authenticate success, global_id 4597 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 10 monclient: _renew_subs 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4520143910 con 0x7f452013c480 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 10 client.4597.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 10 monclient: _renew_subs 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4520130ad0 con 0x7f452013c480 2024-03-25T18:23:04.493 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.490+0000 7f4525406640 1 librados: init done 2024-03-25T18:23:04.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.491+0000 7f4525406640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:04.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.491+0000 7f4525406640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:04.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.491+0000 7f4525406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:04.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.491+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f452006dbd0 con 0x7f452013c480 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.517+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4518002070 con 0x7f452013c480 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.518+0000 7f45077fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.518+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f4518003b10 con 0x7f452013c480 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.518+0000 7f45077fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.518+0000 7f45077fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.518+0000 7f45077fe640 10 monclient: dump: 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:04.519 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:04.520 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:04.520 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:04.520 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:04.520 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:04.520 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:04.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.518+0000 7f451d7fa640 4 set_mon_vals no callback set 2024-03-25T18:23:04.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.518+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f4518003d30 con 0x7f452013c480 2024-03-25T18:23:04.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.519+0000 7f451effd640 10 monclient: get_auth_request con 0x7f450c06de60 auth_method 0 2024-03-25T18:23:04.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.519+0000 7f45077fe640 10 client.4597.objecter ms_handle_connect 0x7f450c06de60 2024-03-25T18:23:04.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.522+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f45180537f0 con 0x7f452013c480 2024-03-25T18:23:04.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.522+0000 7f45077fe640 10 client.4597.objecter ms_dispatch 0x7f4520000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:04.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.522+0000 7f45077fe640 3 client.4597.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:04.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.522+0000 7f45077fe640 3 client.4597.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:04.523 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.522+0000 7f45077fe640 20 client.4597.objecter dump_active .. 0 homeless 2024-03-25T18:23:04.783 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.782+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f451804f300 con 0x7f452013c480 2024-03-25T18:23:04.783 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.782+0000 7f45077fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:04.783 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:04.782+0000 7f45077fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:05.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.003+0000 7f4525406640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.003+0000 7f4525406640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.003+0000 7f4525406640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:05.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.003+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4520143aa0 con 0x7f452013c480 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.056+0000 7f45077fe640 1 -- 172.21.15.7:0/2088609362 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f4518051230 con 0x7f452013c480 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.056+0000 7f45077fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.056+0000 7f45077fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:05.058 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:05.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 10 librados: watch_flush enter 2024-03-25T18:23:05.060 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 10 librados: watch_flush exit 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 20 client.4597.objecter shutdown clearing up homeless session... 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 10 client.4597.objecter successfully canceled tick 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 >> v1:172.21.15.7:6800/1717564767 conn(0x7f450c06de60 legacy=0x7f450c070320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 10 monclient: shutdown 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 >> v1:172.21.15.183:6789/0 conn(0x7f452013c480 legacy=0x7f45201303c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 shutdown_connections 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 >> 172.21.15.7:0/2088609362 conn(0x7f45200617a0 msgr2=0x7f4520140760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 shutdown_connections 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 1 -- 172.21.15.7:0/2088609362 wait complete. 2024-03-25T18:23:05.061 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:05.059+0000 7f4525406640 1 librados: shutdown 2024-03-25T18:23:05.072 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:11.879 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:11.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.995+0000 7f2129396640 10 monclient: get_monmap_and_config 2024-03-25T18:23:11.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.995+0000 7f2129396640 10 monclient: build_initial_monmap 2024-03-25T18:23:11.997 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.995+0000 7f2129396640 10 monclient: monmap: 2024-03-25T18:23:11.997 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:11.997 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:11.997 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:11.996975+0000 2024-03-25T18:23:11.997 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:11.996975+0000 2024-03-25T18:23:11.998 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:11.998 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:11.998 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:11.998 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:11.998 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:11.998 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.996+0000 7f2129396640 1 Processor -- start 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.996+0000 7f2129396640 1 -- start start 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.996+0000 7f2129396640 10 monclient: init 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 10 monclient(hunting): picked mon.noname-b con 0x7f21240b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 10 monclient(hunting): picked mon.noname-c con 0x7f21240b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 10 monclient(hunting): picked mon.noname-a con 0x7f212413c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:11.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21240ab0e0 con 0x7f212413c480 2024-03-25T18:23:12.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21240ab3c0 con 0x7f21240b4670 2024-03-25T18:23:12.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21240a9f30 con 0x7f21240b4ff0 2024-03-25T18:23:12.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:12.000 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:11.997+0000 7f2129396640 10 monclient(hunting): authenticate will time out at 7834.102299s 2024-03-25T18:23:12.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.008+0000 7f2122ffd640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f21240b4ff0 0x7f212413bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:33230/0 (socket says 172.21.15.7:33230) 2024-03-25T18:23:12.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.008+0000 7f2122ffd640 1 -- 172.21.15.7:0/3185849351 learned_addr learned my addr 172.21.15.7:0/3185849351 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:12.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.018+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 531862677 0 0) 0x7f21240a9f30 con 0x7f21240b4ff0 2024-03-25T18:23:12.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.018+0000 7f2121ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.018+0000 7f2121ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.018+0000 7f2121ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.018+0000 7f2121ffb640 10 monclient(hunting): my global_id is 4631 2024-03-25T18:23:12.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.018+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f210c003520 con 0x7f21240b4ff0 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3696137506 0 0) 0x7f210c003520 con 0x7f21240b4ff0 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 >> v1:172.21.15.183:6789/0 conn(0x7f21240b4670 legacy=0x7f21241425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 >> v1:172.21.15.7:6789/0 conn(0x7f212413c480 legacy=0x7f212414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 1 monclient: found mon.noname-b 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:23:12.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21240a9b70 con 0x7f21240b4ff0 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 10 monclient: _finish_auth 0 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2114001690 con 0x7f21240b4ff0 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2121ffb640 10 monclient: dump: 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:12.032 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:12.033 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:12.033 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:12.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2129396640 5 monclient: authenticate success, global_id 4631 2024-03-25T18:23:12.033 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.030+0000 7f2129396640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f21140017f0 con 0x7f21240b4ff0 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2121ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2121ffb640 1 -- 172.21.15.7:0/3185849351 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f2114001c40 con 0x7f21240b4ff0 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2129396640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2129396640 4 set_mon_vals no callback set 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2121ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2121ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.046+0000 7f2121ffb640 10 monclient: dump: 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.048 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:12.049 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:12.049 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:12.049 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:12.049 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.047+0000 7f2129396640 10 monclient: shutdown 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.047+0000 7f2129396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.047+0000 7f2129396640 1 -- 172.21.15.7:0/3185849351 >> v1:172.21.15.7:6790/0 conn(0x7f21240b4ff0 legacy=0x7f212413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.047+0000 7f2129396640 1 -- 172.21.15.7:0/3185849351 shutdown_connections 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.047+0000 7f2129396640 1 -- 172.21.15.7:0/3185849351 >> 172.21.15.7:0/3185849351 conn(0x7f21240617a0 msgr2=0x7f2124061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.047+0000 7f2129396640 1 -- 172.21.15.7:0/3185849351 shutdown_connections 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.047+0000 7f2129396640 1 -- 172.21.15.7:0/3185849351 wait complete. 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.048+0000 7f2129396640 10 monclient: build_initial_monmap 2024-03-25T18:23:12.050 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.048+0000 7f2129396640 10 monclient: monmap: 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.048+0000 7f2129396640 1 librados: starting msgr at 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.048+0000 7f2129396640 1 librados: starting objecter 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.049+0000 7f2129396640 1 Processor -- start 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.049+0000 7f2129396640 1 -- start start 2024-03-25T18:23:12.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.049+0000 7f2129396640 1 librados: setting wanted keys 2024-03-25T18:23:12.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.049+0000 7f2129396640 1 librados: calling monclient init 2024-03-25T18:23:12.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.049+0000 7f2129396640 10 monclient: init 2024-03-25T18:23:12.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:12.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 10 monclient(hunting): picked mon.noname-a con 0x7f21240b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 10 monclient(hunting): picked mon.noname-c con 0x7f21240b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 10 monclient(hunting): picked mon.noname-b con 0x7f212413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212413d2f0 con 0x7f21240b4670 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2124143810 con 0x7f212413c480 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f212413e190 con 0x7f21240b4ff0 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:12.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.050+0000 7f2129396640 10 monclient(hunting): authenticate will time out at 7834.154882s 2024-03-25T18:23:12.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.062+0000 7f21237fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f212413c480 0x7f21241e2750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:45254/0 (socket says 172.21.15.7:45254) 2024-03-25T18:23:12.063 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.062+0000 7f21237fe640 1 -- 172.21.15.7:0/2228991921 learned_addr learned my addr 172.21.15.7:0/2228991921 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:12.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.066+0000 7f210b7fe640 10 client.?.objecter ms_handle_connect 0x7f212413c480 2024-03-25T18:23:12.067 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.066+0000 7f210b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.076+0000 7f210b7fe640 10 client.?.objecter ms_handle_connect 0x7f21240b4ff0 2024-03-25T18:23:12.077 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.076+0000 7f210b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.079+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850465429 0 0) 0x7f2124143810 con 0x7f212413c480 2024-03-25T18:23:12.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.079+0000 7f210b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.079+0000 7f210b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.079+0000 7f210b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.079+0000 7f210b7fe640 10 monclient(hunting): my global_id is 4603 2024-03-25T18:23:12.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.079+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2110004eb0 con 0x7f212413c480 2024-03-25T18:23:12.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.082+0000 7f210b7fe640 10 client.?.objecter ms_handle_connect 0x7f21240b4670 2024-03-25T18:23:12.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.082+0000 7f210b7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:12.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.087+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710404111 0 0) 0x7f212413e190 con 0x7f21240b4ff0 2024-03-25T18:23:12.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.087+0000 7f210b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.087+0000 7f210b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.088 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.087+0000 7f210b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.087+0000 7f210b7fe640 10 monclient(hunting): my global_id is 4634 2024-03-25T18:23:12.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.087+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2124143810 con 0x7f21240b4ff0 2024-03-25T18:23:12.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.091+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2402585616 0 0) 0x7f2110004eb0 con 0x7f212413c480 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.091+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f212413e190 con 0x7f212413c480 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.091+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f211c001c30 con 0x7f212413c480 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.091+0000 7f210b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.091+0000 7f210b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.091+0000 7f210b7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:12.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.091+0000 7f210b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:12.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.094+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1453714715 0 0) 0x7f212413d2f0 con 0x7f21240b4670 2024-03-25T18:23:12.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.094+0000 7f210b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:12.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.094+0000 7f210b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:12.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.094+0000 7f210b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:12.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.094+0000 7f210b7fe640 10 monclient(hunting): my global_id is 4599 2024-03-25T18:23:12.095 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.094+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2110004eb0 con 0x7f21240b4670 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 503904580 0 0) 0x7f212413e190 con 0x7f212413c480 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 >> v1:172.21.15.7:6790/0 conn(0x7f21240b4ff0 legacy=0x7f212413ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 >> v1:172.21.15.7:6789/0 conn(0x7f21240b4670 legacy=0x7f212413cbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 1 monclient: found mon.b 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212413e470 con 0x7f212413c480 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:12.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f210b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 5 monclient: authenticate success, global_id 4603 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 10 monclient: _renew_subs 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2124145e80 con 0x7f212413c480 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 10 client.4603.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 10 monclient: _renew_subs 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f212413f800 con 0x7f212413c480 2024-03-25T18:23:12.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.097+0000 7f2129396640 1 librados: init done 2024-03-25T18:23:12.101 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.099+0000 7f2129396640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.101 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.099+0000 7f2129396640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.101 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.099+0000 7f2129396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:12.101 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.099+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f212413e890 con 0x7f212413c480 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.377+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f211c003990 con 0x7f212413c480 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.377+0000 7f210b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.377+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f211c003b80 con 0x7f212413c480 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.377+0000 7f210b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.377+0000 7f210b7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.377+0000 7f210b7fe640 10 monclient: dump: 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:12.379 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.377+0000 7f21217fa640 4 set_mon_vals no callback set 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.378+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f211c003da0 con 0x7f212413c480 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.379+0000 7f2122ffd640 10 monclient: get_auth_request con 0x7f211006de00 auth_method 0 2024-03-25T18:23:12.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.379+0000 7f210b7fe640 10 client.4603.objecter ms_handle_connect 0x7f211006de00 2024-03-25T18:23:12.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f211c0548b0 con 0x7f212413c480 2024-03-25T18:23:12.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 10 client.4603.objecter ms_dispatch 0x7f2124000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:12.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 3 client.4603.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:12.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 3 client.4603.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:12.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 20 client.4603.objecter dump_active .. 0 homeless 2024-03-25T18:23:12.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f211c0503c0 con 0x7f212413c480 2024-03-25T18:23:12.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:12.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.383+0000 7f210b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:12.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.644+0000 7f2129396640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:12.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.644+0000 7f2129396640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:12.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.644+0000 7f2129396640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:12.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.644+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2124143410 con 0x7f212413c480 2024-03-25T18:23:12.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.672+0000 7f210b7fe640 1 -- 172.21.15.7:0/2228991921 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f211c0522f0 con 0x7f212413c480 2024-03-25T18:23:12.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.672+0000 7f210b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:12.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.672+0000 7f210b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:12.674 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:12.674 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 10 librados: watch_flush enter 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 10 librados: watch_flush exit 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 20 client.4603.objecter shutdown clearing up homeless session... 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 10 client.4603.objecter successfully canceled tick 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 >> v1:172.21.15.7:6800/1717564767 conn(0x7f211006de00 legacy=0x7f21100702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 10 monclient: shutdown 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:12.676 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 >> v1:172.21.15.183:6789/0 conn(0x7f212413c480 legacy=0x7f21241e2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:12.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 shutdown_connections 2024-03-25T18:23:12.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 >> 172.21.15.7:0/2228991921 conn(0x7f21240617a0 msgr2=0x7f21241407a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:12.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 shutdown_connections 2024-03-25T18:23:12.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.674+0000 7f2129396640 1 -- 172.21.15.7:0/2228991921 wait complete. 2024-03-25T18:23:12.677 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:12.675+0000 7f2129396640 1 librados: shutdown 2024-03-25T18:23:12.687 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:14.193 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:20.830 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.201 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:20.838 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.238 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:20.875 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.253 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:20.890 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.257 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:20.883 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.261 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:20.887 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.265 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:20.902 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.285 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:20.911 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:14.295 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:20.933 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:19.493 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.609+0000 7efe52664640 10 monclient: get_monmap_and_config 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.609+0000 7efe52664640 10 monclient: build_initial_monmap 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.610+0000 7efe52664640 10 monclient: monmap: 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:19.611006+0000 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:19.611006+0000 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:19.611 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:19.612 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:19.612 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.610+0000 7efe52664640 1 Processor -- start 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.610+0000 7efe52664640 1 -- start start 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.610+0000 7efe52664640 10 monclient: init 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 10 monclient(hunting): picked mon.noname-b con 0x7efe4c0b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 10 monclient(hunting): picked mon.noname-c con 0x7efe4c0b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 10 monclient(hunting): picked mon.noname-a con 0x7efe4c0b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c0a96f0 con 0x7efe4c0b1b00 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c0ab0e0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.613 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c145f70 con 0x7efe4c0b4e70 2024-03-25T18:23:19.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:19.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe52664640 10 monclient(hunting): authenticate will time out at 7841.716137s 2024-03-25T18:23:19.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe4b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7efe4c0b44b0 0x7efe4c145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:49780/0 (socket says 172.21.15.7:49780) 2024-03-25T18:23:19.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.611+0000 7efe4b7fe640 1 -- 172.21.15.7:0/2168240202 learned_addr learned my addr 172.21.15.7:0/2168240202 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:19.698 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.697+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1989598335 0 0) 0x7efe4c0a96f0 con 0x7efe4c0b1b00 2024-03-25T18:23:19.698 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.697+0000 7efe4affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.698 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.697+0000 7efe4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.698 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.697+0000 7efe4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.698 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.697+0000 7efe4affd640 10 monclient(hunting): my global_id is 4602 2024-03-25T18:23:19.698 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.697+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe34003520 con 0x7efe4c0b1b00 2024-03-25T18:23:19.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.705+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2509071202 0 0) 0x7efe4c145f70 con 0x7efe4c0b4e70 2024-03-25T18:23:19.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.705+0000 7efe4affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.705+0000 7efe4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.705+0000 7efe4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.705+0000 7efe4affd640 10 monclient(hunting): my global_id is 4637 2024-03-25T18:23:19.706 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.705+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe4c0a96f0 con 0x7efe4c0b4e70 2024-03-25T18:23:19.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.708+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 477458820 0 0) 0x7efe4c0ab0e0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.708+0000 7efe4affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.708+0000 7efe4affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.708+0000 7efe4affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.708+0000 7efe4affd640 10 monclient(hunting): my global_id is 4606 2024-03-25T18:23:19.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.708+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe4c145f70 con 0x7efe4c0b44b0 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.730+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3329052996 0 0) 0x7efe4c145f70 con 0x7efe4c0b44b0 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.730+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 >> v1:172.21.15.7:6790/0 conn(0x7efe4c0b4e70 legacy=0x7efe4c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.730+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 >> v1:172.21.15.7:6789/0 conn(0x7efe4c0b1b00 legacy=0x7efe4c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.730+0000 7efe4affd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.730+0000 7efe4affd640 1 monclient: found mon.noname-c 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.730+0000 7efe4affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c0a9b80 con 0x7efe4c0b44b0 2024-03-25T18:23:19.732 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 10 monclient: _finish_auth 0 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7efe400015b0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe4affd640 10 monclient: dump: 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:19.733 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:19.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe52664640 5 monclient: authenticate success, global_id 4606 2024-03-25T18:23:19.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.731+0000 7efe52664640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe40001710 con 0x7efe4c0b44b0 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe4affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe4affd640 1 -- 172.21.15.7:0/2168240202 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7efe40001b60 con 0x7efe4c0b44b0 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe52664640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe52664640 4 set_mon_vals no callback set 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe4affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe4affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:19.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.745+0000 7efe4affd640 10 monclient: dump: 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:19.747 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:19.748 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.746+0000 7efe52664640 10 monclient: shutdown 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.746+0000 7efe52664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.746+0000 7efe52664640 1 -- 172.21.15.7:0/2168240202 >> v1:172.21.15.183:6789/0 conn(0x7efe4c0b44b0 legacy=0x7efe4c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.746+0000 7efe52664640 1 -- 172.21.15.7:0/2168240202 shutdown_connections 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.746+0000 7efe52664640 1 -- 172.21.15.7:0/2168240202 >> 172.21.15.7:0/2168240202 conn(0x7efe4c0617a0 msgr2=0x7efe4c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.746+0000 7efe52664640 1 -- 172.21.15.7:0/2168240202 shutdown_connections 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.746+0000 7efe52664640 1 -- 172.21.15.7:0/2168240202 wait complete. 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 10 monclient: build_initial_monmap 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 10 monclient: monmap: 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:19.749 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 1 librados: starting msgr at 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 1 librados: starting objecter 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 1 Processor -- start 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 1 -- start start 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 1 librados: setting wanted keys 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 1 librados: calling monclient init 2024-03-25T18:23:19.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.747+0000 7efe52664640 10 monclient: init 2024-03-25T18:23:19.751 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 10 monclient(hunting): picked mon.noname-a con 0x7efe4c0b1b00 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 10 monclient(hunting): picked mon.noname-b con 0x7efe4c0b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 10 monclient(hunting): picked mon.noname-c con 0x7efe4c0b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c1357e0 con 0x7efe4c0b1b00 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c1399f0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe4c139b80 con 0x7efe4c0b4e70 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:19.752 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.748+0000 7efe52664640 10 monclient(hunting): authenticate will time out at 7841.853482s 2024-03-25T18:23:19.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.764+0000 7efe50c01640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7efe4c0b4e70 0x7efe4c1e2480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:37720/0 (socket says 172.21.15.7:37720) 2024-03-25T18:23:19.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.764+0000 7efe50c01640 1 -- 172.21.15.7:0/2398256876 learned_addr learned my addr 172.21.15.7:0/2398256876 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:19.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.766+0000 7efe48ff9640 10 client.?.objecter ms_handle_connect 0x7efe4c0b4e70 2024-03-25T18:23:19.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.766+0000 7efe48ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:19.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.770+0000 7efe48ff9640 10 client.?.objecter ms_handle_connect 0x7efe4c0b44b0 2024-03-25T18:23:19.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.770+0000 7efe48ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:19.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.777+0000 7efe48ff9640 10 client.?.objecter ms_handle_connect 0x7efe4c0b1b00 2024-03-25T18:23:19.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.777+0000 7efe48ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:19.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.778+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 550545822 0 0) 0x7efe4c1399f0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.778+0000 7efe48ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.778+0000 7efe48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.778+0000 7efe48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.778+0000 7efe48ff9640 10 monclient(hunting): my global_id is 4609 2024-03-25T18:23:19.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.778+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38004eb0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.791 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1946911341 0 0) 0x7efe4c139b80 con 0x7efe4c0b4e70 2024-03-25T18:23:19.791 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): my global_id is 4640 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe4c1399f0 con 0x7efe4c0b4e70 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1410363092 0 0) 0x7efe38004eb0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe4c139b80 con 0x7efe4c0b44b0 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7efe40001a20 con 0x7efe4c0b44b0 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.792 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.793 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:19.793 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:19.793 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:19.793 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:19.793 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:19.793 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.790+0000 7efe48ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:19.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.799+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3488304451 0 0) 0x7efe4c1357e0 con 0x7efe4c0b1b00 2024-03-25T18:23:19.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.799+0000 7efe48ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:19.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.799+0000 7efe48ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:19.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.799+0000 7efe48ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:19.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.799+0000 7efe48ff9640 10 monclient(hunting): my global_id is 4605 2024-03-25T18:23:19.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.799+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efe38004eb0 con 0x7efe4c0b1b00 2024-03-25T18:23:19.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.813+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2296937532 0 0) 0x7efe4c1399f0 con 0x7efe4c0b4e70 2024-03-25T18:23:19.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.813+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efe4c1357e0 con 0x7efe4c0b4e70 2024-03-25T18:23:19.814 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.813+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7efe44001c50 con 0x7efe4c0b4e70 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.813+0000 7efe48ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.813+0000 7efe48ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.813+0000 7efe48ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:19.815 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 672319651 0 0) 0x7efe4c139b80 con 0x7efe4c0b44b0 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 >> v1:172.21.15.7:6790/0 conn(0x7efe4c0b4e70 legacy=0x7efe4c1e2480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 >> v1:172.21.15.7:6789/0 conn(0x7efe4c0b1b00 legacy=0x7efe4c1d48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 1 monclient: found mon.b 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c142fa0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:19.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe48ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 5 monclient: authenticate success, global_id 4609 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 10 monclient: _renew_subs 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe4c1431f0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 10 client.4609.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 10 monclient: _renew_subs 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efe4c1353a0 con 0x7efe4c0b44b0 2024-03-25T18:23:19.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.818+0000 7efe52664640 1 librados: init done 2024-03-25T18:23:19.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.819+0000 7efe52664640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:19.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.819+0000 7efe52664640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:19.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.819+0000 7efe52664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:19.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:19.819+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe4c0b7b90 con 0x7efe4c0b44b0 2024-03-25T18:23:20.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.096+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efe40004a90 con 0x7efe4c0b44b0 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.096+0000 7efe48ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.096+0000 7efe4a7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.096+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7efe40004da0 con 0x7efe4c0b44b0 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.096+0000 7efe48ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.096+0000 7efe48ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.096+0000 7efe48ff9640 10 monclient: dump: 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:20.098 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:20.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.097+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7efe4001ad70 con 0x7efe4c0b44b0 2024-03-25T18:23:20.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.098+0000 7efe4bfff640 10 monclient: get_auth_request con 0x7efe3806df60 auth_method 0 2024-03-25T18:23:20.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.098+0000 7efe48ff9640 10 client.4609.objecter ms_handle_connect 0x7efe3806df60 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.101+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7efe400548a0 con 0x7efe4c0b44b0 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.101+0000 7efe48ff9640 10 client.4609.objecter ms_dispatch 0x7efe4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.101+0000 7efe48ff9640 3 client.4609.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.101+0000 7efe48ff9640 3 client.4609.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.101+0000 7efe48ff9640 20 client.4609.objecter dump_active .. 0 homeless 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.102+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efe400503b0 con 0x7efe4c0b44b0 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.102+0000 7efe48ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:20.103 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.102+0000 7efe48ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.356+0000 7efe52664640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.356+0000 7efe52664640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.356+0000 7efe52664640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:20.357 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.356+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efe4c1d45d0 con 0x7efe4c0b44b0 2024-03-25T18:23:20.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.384+0000 7efe48ff9640 1 -- 172.21.15.7:0/2398256876 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7efe4c1d45d0 con 0x7efe4c0b44b0 2024-03-25T18:23:20.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.384+0000 7efe48ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:20.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.384+0000 7efe48ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:20.385 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:20.386 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 10 librados: watch_flush enter 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 10 librados: watch_flush exit 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 20 client.4609.objecter shutdown clearing up homeless session... 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 10 client.4609.objecter successfully canceled tick 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 >> v1:172.21.15.7:6800/1717564767 conn(0x7efe3806df60 legacy=0x7efe38070420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 10 monclient: shutdown 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 >> v1:172.21.15.183:6789/0 conn(0x7efe4c0b44b0 legacy=0x7efe4c1d4fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 shutdown_connections 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 >> 172.21.15.7:0/2398256876 conn(0x7efe4c0617a0 msgr2=0x7efe4c0b6660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 shutdown_connections 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 1 -- 172.21.15.7:0/2398256876 wait complete. 2024-03-25T18:23:20.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:20.386+0000 7efe52664640 1 librados: shutdown 2024-03-25T18:23:20.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:27.207 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.323+0000 7f72b1602640 10 monclient: get_monmap_and_config 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.323+0000 7f72b1602640 10 monclient: build_initial_monmap 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.323+0000 7f72b1602640 10 monclient: monmap: 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:27.324253+0000 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:27.324253+0000 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:27.324 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:27.325 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:27.325 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:27.325 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 1 Processor -- start 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 1 -- start start 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient: init 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient(hunting): picked mon.noname-a con 0x7f72ac0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient(hunting): picked mon.noname-c con 0x7f72ac0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient(hunting): picked mon.noname-b con 0x7f72ac13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0ab0e0 con 0x7f72ac0b4ff0 2024-03-25T18:23:27.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0ab3c0 con 0x7f72ac0b4670 2024-03-25T18:23:27.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac0a9f30 con 0x7f72ac13c480 2024-03-25T18:23:27.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:27.327 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.324+0000 7f72b1602640 10 monclient(hunting): authenticate will time out at 7849.429436s 2024-03-25T18:23:27.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.328+0000 7f72ab7fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f72ac13c480 0x7f72ac14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:48580/0 (socket says 172.21.15.7:48580) 2024-03-25T18:23:27.329 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.328+0000 7f72ab7fe640 1 -- 172.21.15.7:0/58054688 learned_addr learned my addr 172.21.15.7:0/58054688 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:27.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.337+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2370355520 0 0) 0x7f72ac0a9f30 con 0x7f72ac13c480 2024-03-25T18:23:27.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.337+0000 7f72aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.337+0000 7f72aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.337+0000 7f72aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.337+0000 7f72aa7fc640 10 monclient(hunting): my global_id is 4643 2024-03-25T18:23:27.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.337+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7298003520 con 0x7f72ac13c480 2024-03-25T18:23:27.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.347+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389280944 0 0) 0x7f72ac0ab0e0 con 0x7f72ac0b4ff0 2024-03-25T18:23:27.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.347+0000 7f72aa7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.347+0000 7f72aa7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.347+0000 7f72aa7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.347+0000 7f72aa7fc640 10 monclient(hunting): my global_id is 4608 2024-03-25T18:23:27.349 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.347+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac0a9f30 con 0x7f72ac0b4ff0 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1009508507 0 0) 0x7f7298003520 con 0x7f72ac13c480 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 >> v1:172.21.15.183:6789/0 conn(0x7f72ac0b4670 legacy=0x7f72ac1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 >> v1:172.21.15.7:6789/0 conn(0x7f72ac0b4ff0 legacy=0x7f72ac13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 1 monclient: found mon.noname-b 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac0a9b70 con 0x7f72ac13c480 2024-03-25T18:23:27.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 10 monclient: _finish_auth 0 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f72a4001a10 con 0x7f72ac13c480 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72aa7fc640 10 monclient: dump: 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:27.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72b1602640 5 monclient: authenticate success, global_id 4643 2024-03-25T18:23:27.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.350+0000 7f72b1602640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72a4001b70 con 0x7f72ac13c480 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72aa7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72aa7fc640 1 -- 172.21.15.7:0/58054688 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f72a4001f80 con 0x7f72ac13c480 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72b1602640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72b1602640 4 set_mon_vals no callback set 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72aa7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72aa7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.369+0000 7f72aa7fc640 10 monclient: dump: 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.371 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:27.372 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:27.372 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:27.372 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:27.372 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.370+0000 7f72b1602640 10 monclient: shutdown 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.370+0000 7f72b1602640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.370+0000 7f72b1602640 1 -- 172.21.15.7:0/58054688 >> v1:172.21.15.7:6790/0 conn(0x7f72ac13c480 legacy=0x7f72ac14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.370+0000 7f72b1602640 1 -- 172.21.15.7:0/58054688 shutdown_connections 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.370+0000 7f72b1602640 1 -- 172.21.15.7:0/58054688 >> 172.21.15.7:0/58054688 conn(0x7f72ac0617a0 msgr2=0x7f72ac061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.370+0000 7f72b1602640 1 -- 172.21.15.7:0/58054688 shutdown_connections 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.370+0000 7f72b1602640 1 -- 172.21.15.7:0/58054688 wait complete. 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 10 monclient: build_initial_monmap 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 10 monclient: monmap: 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:27.374 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 1 librados: starting msgr at 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 1 librados: starting objecter 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 1 Processor -- start 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 1 -- start start 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 1 librados: setting wanted keys 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 1 librados: calling monclient init 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.371+0000 7f72b1602640 10 monclient: init 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:27.375 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 10 monclient(hunting): picked mon.noname-b con 0x7f72ac0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 10 monclient(hunting): picked mon.noname-a con 0x7f72ac0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 10 monclient(hunting): picked mon.noname-c con 0x7f72ac13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac146560 con 0x7f72ac0b4ff0 2024-03-25T18:23:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac1db970 con 0x7f72ac0b4670 2024-03-25T18:23:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72ac1db520 con 0x7f72ac13c480 2024-03-25T18:23:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:27.376 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.372+0000 7f72b1602640 10 monclient(hunting): authenticate will time out at 7849.477290s 2024-03-25T18:23:27.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.381+0000 7f72aaffd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f72ac0b4670 0x7f72ac1dbe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:56848/0 (socket says 172.21.15.7:56848) 2024-03-25T18:23:27.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.382+0000 7f72aaffd640 1 -- 172.21.15.7:0/173407645 learned_addr learned my addr 172.21.15.7:0/173407645 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:27.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.383+0000 7f72a37fe640 10 client.?.objecter ms_handle_connect 0x7f72ac0b4670 2024-03-25T18:23:27.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.383+0000 7f72a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.388+0000 7f72a37fe640 10 client.?.objecter ms_handle_connect 0x7f72ac0b4ff0 2024-03-25T18:23:27.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.388+0000 7f72a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.389+0000 7f72a37fe640 10 client.?.objecter ms_handle_connect 0x7f72ac13c480 2024-03-25T18:23:27.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.389+0000 7f72a37fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:27.396 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.395+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125669194 0 0) 0x7f72ac1db970 con 0x7f72ac0b4670 2024-03-25T18:23:27.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.395+0000 7f72a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.395+0000 7f72a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.395+0000 7f72a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.395+0000 7f72a37fe640 10 monclient(hunting): my global_id is 4612 2024-03-25T18:23:27.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.395+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f728c004eb0 con 0x7f72ac0b4670 2024-03-25T18:23:27.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.402+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 515633160 0 0) 0x7f72ac1db520 con 0x7f72ac13c480 2024-03-25T18:23:27.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): my global_id is 4646 2024-03-25T18:23:27.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac1db970 con 0x7f72ac13c480 2024-03-25T18:23:27.404 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4106454061 0 0) 0x7f72ac146560 con 0x7f72ac0b4ff0 2024-03-25T18:23:27.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:27.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:27.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:27.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 10 monclient(hunting): my global_id is 4611 2024-03-25T18:23:27.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.403+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac1db520 con 0x7f72ac0b4ff0 2024-03-25T18:23:27.411 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.410+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4084267025 0 0) 0x7f728c004eb0 con 0x7f72ac0b4670 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.410+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72ac146560 con 0x7f72ac0b4670 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.410+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f729c001a90 con 0x7f72ac0b4670 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.410+0000 7f72a37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.410+0000 7f72a37fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.410+0000 7f72a37fe640 10 monclient(hunting): dump: 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:27.412 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:27.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.410+0000 7f72a37fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 989767483 0 0) 0x7f72ac146560 con 0x7f72ac0b4670 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 >> v1:172.21.15.7:6790/0 conn(0x7f72ac13c480 legacy=0x7f72ac145e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 >> v1:172.21.15.7:6789/0 conn(0x7f72ac0b4ff0 legacy=0x7f72ac145740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 1 monclient: found mon.b 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72ac1db6b0 con 0x7f72ac0b4670 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72a37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 5 monclient: authenticate success, global_id 4612 2024-03-25T18:23:27.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 10 monclient: _renew_subs 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac13f610 con 0x7f72ac0b4670 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 10 client.4612.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 10 monclient: _renew_subs 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.419+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72ac143920 con 0x7f72ac0b4670 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.420+0000 7f72b1602640 1 librados: init done 2024-03-25T18:23:27.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.420+0000 7f72b1602640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.420+0000 7f72b1602640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.420+0000 7f72b1602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:27.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.420+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72ac1dbb00 con 0x7f72ac0b4670 2024-03-25T18:23:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.440+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f729c001cd0 con 0x7f72ac0b4670 2024-03-25T18:23:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.440+0000 7f72a37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.440+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f729c003a60 con 0x7f72ac0b4670 2024-03-25T18:23:27.441 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.440+0000 7f72a37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.440+0000 7f72a37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.440+0000 7f72a37fe640 10 monclient: dump: 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:27.442 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.440+0000 7f72a9ffb640 4 set_mon_vals no callback set 2024-03-25T18:23:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.441+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f729c003c80 con 0x7f72ac0b4670 2024-03-25T18:23:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.441+0000 7f72a3fff640 10 monclient: get_auth_request con 0x7f728c06ddf0 auth_method 0 2024-03-25T18:23:27.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.441+0000 7f72a37fe640 10 client.4612.objecter ms_handle_connect 0x7f728c06ddf0 2024-03-25T18:23:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.443+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f729c054890 con 0x7f72ac0b4670 2024-03-25T18:23:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.443+0000 7f72a37fe640 10 client.4612.objecter ms_dispatch 0x7f72ac000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.443+0000 7f72a37fe640 3 client.4612.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:27.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.443+0000 7f72a37fe640 3 client.4612.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:27.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.443+0000 7f72a37fe640 20 client.4612.objecter dump_active .. 0 homeless 2024-03-25T18:23:27.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.700+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f729c0503a0 con 0x7f72ac0b4670 2024-03-25T18:23:27.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.700+0000 7f72a37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:27.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.700+0000 7f72a37fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:27.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.935+0000 7f72b1602640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:27.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.935+0000 7f72b1602640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:27.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.935+0000 7f72b1602640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:27.936 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.935+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72ac06dbd0 con 0x7f72ac0b4670 2024-03-25T18:23:27.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.955+0000 7f72a37fe640 1 -- 172.21.15.7:0/173407645 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f729c0522d0 con 0x7f72ac0b4670 2024-03-25T18:23:27.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.955+0000 7f72a37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:27.956 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.955+0000 7f72a37fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:27.956 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:27.956 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 10 librados: watch_flush enter 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 10 librados: watch_flush exit 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 20 client.4612.objecter shutdown clearing up homeless session... 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 10 client.4612.objecter successfully canceled tick 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 >> v1:172.21.15.7:6800/1717564767 conn(0x7f728c06ddf0 legacy=0x7f728c0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 10 monclient: shutdown 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 >> v1:172.21.15.183:6789/0 conn(0x7f72ac0b4670 legacy=0x7f72ac1dbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 shutdown_connections 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 >> 172.21.15.7:0/173407645 conn(0x7f72ac0617a0 msgr2=0x7f72ac0b8690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:27.959 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.957+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 shutdown_connections 2024-03-25T18:23:27.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.958+0000 7f72b1602640 1 -- 172.21.15.7:0/173407645 wait complete. 2024-03-25T18:23:27.960 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:27.958+0000 7f72b1602640 1 librados: shutdown 2024-03-25T18:23:27.970 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:34.244 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:40.881 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.256 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:40.893 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.285 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:09:40.922 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.324 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:40.950 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.327 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:40.953 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.330 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:40.967 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.341 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:40.978 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.347 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:09:40.973 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.380 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:09:41.017 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:34.777 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:34.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.894+0000 7f5a1d214640 10 monclient: get_monmap_and_config 2024-03-25T18:23:34.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.894+0000 7f5a1d214640 10 monclient: build_initial_monmap 2024-03-25T18:23:34.895 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.894+0000 7f5a1d214640 10 monclient: monmap: 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:34.895549+0000 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:34.895549+0000 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:34.896 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:34.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 1 Processor -- start 2024-03-25T18:23:34.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 1 -- start start 2024-03-25T18:23:34.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient: init 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient(hunting): picked mon.noname-c con 0x7f5a180b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient(hunting): picked mon.noname-a con 0x7f5a180b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient(hunting): picked mon.noname-b con 0x7f5a1813c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180ab0e0 con 0x7f5a180b4670 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180ab3c0 con 0x7f5a180b4ff0 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180a9f30 con 0x7f5a1813c480 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:34.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.895+0000 7f5a1d214640 10 monclient(hunting): authenticate will time out at 7857.000491s 2024-03-25T18:23:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.902+0000 7f5a1759e640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f5a1813c480 0x7f5a1814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:43356/0 (socket says 172.21.15.7:43356) 2024-03-25T18:23:34.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.902+0000 7f5a1759e640 1 -- 172.21.15.7:0/2314870309 learned_addr learned my addr 172.21.15.7:0/2314870309 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:34.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2223409240 0 0) 0x7f5a180a9f30 con 0x7f5a1813c480 2024-03-25T18:23:34.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:34.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:34.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): my global_id is 4649 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a00003520 con 0x7f5a1813c480 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4168903389 0 0) 0x7f5a180ab3c0 con 0x7f5a180b4ff0 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): _negotiate 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 10 monclient(hunting): my global_id is 4615 2024-03-25T18:23:34.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.917+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a180a9f30 con 0x7f5a180b4ff0 2024-03-25T18:23:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1038028438 0 0) 0x7f5a180a9f30 con 0x7f5a180b4ff0 2024-03-25T18:23:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 >> v1:172.21.15.7:6790/0 conn(0x7f5a1813c480 legacy=0x7f5a1814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 >> v1:172.21.15.7:6789/0 conn(0x7f5a180b4670 legacy=0x7f5a181425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.928 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 10 monclient: _finish_hunting 0 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 1 monclient: found mon.noname-c 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a180a9b70 con 0x7f5a180b4ff0 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 10 monclient: _finish_auth 0 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 10 monclient: _check_auth_tickets 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a08001670 con 0x7f5a180b4ff0 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a15d9b640 10 monclient: dump: 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:34.929 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:34.930 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:34.930 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:34.930 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:34.930 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:34.930 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:34.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a1d214640 5 monclient: authenticate success, global_id 4615 2024-03-25T18:23:34.930 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.927+0000 7f5a1d214640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.943+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a080017d0 con 0x7f5a180b4ff0 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.943+0000 7f5a15d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.943+0000 7f5a15d9b640 1 -- 172.21.15.7:0/2314870309 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a08001be0 con 0x7f5a180b4ff0 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.943+0000 7f5a1d214640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.943+0000 7f5a1d214640 4 set_mon_vals no callback set 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.943+0000 7f5a15d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a15d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a15d9b640 10 monclient: dump: 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:34.945 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:34.946 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:34.946 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:34.946 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a1d214640 10 monclient: shutdown 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a1d214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a1d214640 1 -- 172.21.15.7:0/2314870309 >> v1:172.21.15.183:6789/0 conn(0x7f5a180b4ff0 legacy=0x7f5a1813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a1d214640 1 -- 172.21.15.7:0/2314870309 shutdown_connections 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a1d214640 1 -- 172.21.15.7:0/2314870309 >> 172.21.15.7:0/2314870309 conn(0x7f5a180617a0 msgr2=0x7f5a18061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.944+0000 7f5a1d214640 1 -- 172.21.15.7:0/2314870309 shutdown_connections 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.945+0000 7f5a1d214640 1 -- 172.21.15.7:0/2314870309 wait complete. 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.945+0000 7f5a1d214640 10 monclient: build_initial_monmap 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.945+0000 7f5a1d214640 10 monclient: monmap: 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:34.947 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.946+0000 7f5a1d214640 1 librados: starting msgr at 2024-03-25T18:23:34.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.946+0000 7f5a1d214640 1 librados: starting objecter 2024-03-25T18:23:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.946+0000 7f5a1d214640 1 Processor -- start 2024-03-25T18:23:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.946+0000 7f5a1d214640 1 -- start start 2024-03-25T18:23:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.946+0000 7f5a1d214640 1 librados: setting wanted keys 2024-03-25T18:23:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.946+0000 7f5a1d214640 1 librados: calling monclient init 2024-03-25T18:23:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.946+0000 7f5a1d214640 10 monclient: init 2024-03-25T18:23:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:34.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 10 monclient(hunting): picked mon.noname-b con 0x7f5a180b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 10 monclient(hunting): picked mon.noname-a con 0x7f5a180b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 10 monclient(hunting): picked mon.noname-c con 0x7f5a1813c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a181395a0 con 0x7f5a180b4ff0 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a180ac2b0 con 0x7f5a180b4670 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a181d7760 con 0x7f5a1813c480 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:34.950 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.947+0000 7f5a1d214640 10 monclient(hunting): authenticate will time out at 7857.052545s 2024-03-25T18:23:34.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.953+0000 7f5a16d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f5a180b4670 0x7f5a181d7c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:52118/0 (socket says 172.21.15.7:52118) 2024-03-25T18:23:34.954 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.953+0000 7f5a16d9d640 1 -- 172.21.15.7:0/3016882016 learned_addr learned my addr 172.21.15.7:0/3016882016 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:34.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.956+0000 7f59ff7fe640 10 client.?.objecter ms_handle_connect 0x7f5a180b4670 2024-03-25T18:23:34.957 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.956+0000 7f59ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:34.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.960+0000 7f59ff7fe640 10 client.?.objecter ms_handle_connect 0x7f5a1813c480 2024-03-25T18:23:34.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.960+0000 7f59ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:34.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.961+0000 7f5a1759e640 1 -- 172.21.15.7:0/3016882016 >> v1:172.21.15.7:6790/0 conn(0x7f5a1813c480 legacy=0x7f5a1813b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-03-25T18:23:34.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.961+0000 7f5a1759e640 1 -- 172.21.15.7:0/3016882016 >> v1:172.21.15.7:6790/0 conn(0x7f5a1813c480 legacy=0x7f5a1813b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-25T18:23:34.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.961+0000 7f5a1759e640 1 --1- 172.21.15.7:0/3016882016 >> v1:172.21.15.7:6790/0 conn(0x7f5a1813c480 0x7f5a1813b110 :-1 s=OPENED pgs=67572 cs=1 l=1).handle_message read tag failed 2024-03-25T18:23:34.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.961+0000 7f5a1759e640 1 --1- 172.21.15.7:0/3016882016 >> v1:172.21.15.7:6790/0 conn(0x7f5a1813c480 0x7f5a1813b110 :-1 s=OPENED pgs=67572 cs=1 l=1).fault on lossy channel, failing 2024-03-25T18:23:34.963 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.961+0000 7f59ff7fe640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.7:6790/0 2024-03-25T18:23:34.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.967+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968877184 0 0) 0x7f5a180ac2b0 con 0x7f5a180b4670 2024-03-25T18:23:34.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.967+0000 7f59ff7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:23:34.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.967+0000 7f59ff7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:34.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.967+0000 7f59ff7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:34.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.967+0000 7f59ff7fe640 10 monclient(hunting): my global_id is 4618 2024-03-25T18:23:34.969 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.967+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a04004eb0 con 0x7f5a180b4670 2024-03-25T18:23:34.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.970+0000 7f59ff7fe640 10 client.?.objecter ms_handle_connect 0x7f5a180b4ff0 2024-03-25T18:23:34.971 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.970+0000 7f59ff7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.983+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 801396589 0 0) 0x7f5a04004eb0 con 0x7f5a180b4670 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.983+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5a180ac2b0 con 0x7f5a180b4670 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.983+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a08001c20 con 0x7f5a180b4670 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.983+0000 7f59ff7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.983+0000 7f59ff7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.983+0000 7f59ff7fe640 10 monclient(hunting): dump: 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:34.985 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:34.986 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:34.986 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:34.986 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.983+0000 7f59ff7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3258765135 0 0) 0x7f5a180ac2b0 con 0x7f5a180b4670 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 >> v1:172.21.15.7:6790/0 conn(0x7f5a1813c480 legacy=0x7f5a1813b110 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 >> v1:172.21.15.7:6789/0 conn(0x7f5a180b4ff0 legacy=0x7f5a181d47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 1 monclient: found mon.b 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a181d8390 con 0x7f5a180b4670 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 10 monclient: _finish_auth 0 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f59ff7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 5 monclient: authenticate success, global_id 4618 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 10 monclient: _renew_subs 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:34.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a181d4ee0 con 0x7f5a180b4670 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 10 client.4618.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 10 monclient: _renew_subs 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a181d42a0 con 0x7f5a180b4670 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.988+0000 7f5a1d214640 1 librados: init done 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.989+0000 7f5a1d214640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.989+0000 7f5a1d214640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.989+0000 7f5a1d214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:34.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:34.989+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a181d78f0 con 0x7f5a180b4670 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.270+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5a080038a0 con 0x7f5a180b4670 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.270+0000 7f59ff7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.270+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5a08003c30 con 0x7f5a180b4670 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.270+0000 7f59ff7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.270+0000 7f59ff7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.270+0000 7f59ff7fe640 10 monclient: dump: 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:35.271 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:35.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.270+0000 7f5a1559a640 4 set_mon_vals no callback set 2024-03-25T18:23:35.273 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.271+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f5a08003e50 con 0x7f5a180b4670 2024-03-25T18:23:35.273 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.271+0000 7f5a1659c640 10 monclient: get_auth_request con 0x7f5a0406dd00 auth_method 0 2024-03-25T18:23:35.273 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.271+0000 7f59ff7fe640 10 client.4618.objecter ms_handle_connect 0x7f5a0406dd00 2024-03-25T18:23:35.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.276+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f5a08054990 con 0x7f5a180b4670 2024-03-25T18:23:35.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.276+0000 7f59ff7fe640 10 client.4618.objecter ms_dispatch 0x7f5a18000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:35.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.276+0000 7f59ff7fe640 3 client.4618.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:35.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.276+0000 7f59ff7fe640 3 client.4618.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:35.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.276+0000 7f59ff7fe640 20 client.4618.objecter dump_active .. 0 homeless 2024-03-25T18:23:35.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.278+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5a080504a0 con 0x7f5a180b4670 2024-03-25T18:23:35.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.278+0000 7f59ff7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:35.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.278+0000 7f59ff7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:35.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.531+0000 7f5a1d214640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:35.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.531+0000 7f5a1d214640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:35.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.531+0000 7f5a1d214640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:35.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.531+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5a180ac0b0 con 0x7f5a180b4670 2024-03-25T18:23:35.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.613+0000 7f59ff7fe640 1 -- 172.21.15.7:0/3016882016 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f5a080523d0 con 0x7f5a180b4670 2024-03-25T18:23:35.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.613+0000 7f59ff7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:35.614 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.613+0000 7f59ff7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:35.615 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:35.615 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:35.617 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.615+0000 7f5a1d214640 10 librados: watch_flush enter 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 10 librados: watch_flush exit 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 20 client.4618.objecter shutdown clearing up homeless session... 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 10 client.4618.objecter successfully canceled tick 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 >> v1:172.21.15.7:6800/1717564767 conn(0x7f5a0406dd00 legacy=0x7f5a040701c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 10 monclient: shutdown 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 >> v1:172.21.15.183:6789/0 conn(0x7f5a180b4670 legacy=0x7f5a181d7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 shutdown_connections 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 >> 172.21.15.7:0/3016882016 conn(0x7f5a180617a0 msgr2=0x7f5a18140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 shutdown_connections 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 1 -- 172.21.15.7:0/3016882016 wait complete. 2024-03-25T18:23:35.618 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:35.616+0000 7f5a1d214640 1 librados: shutdown 2024-03-25T18:23:35.629 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:42.435 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:42.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.546+0000 7f3538d82640 10 monclient: get_monmap_and_config 2024-03-25T18:23:42.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.546+0000 7f3538d82640 10 monclient: build_initial_monmap 2024-03-25T18:23:42.547 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.546+0000 7f3538d82640 10 monclient: monmap: 2024-03-25T18:23:42.547 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:42.547 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:42.547395+0000 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:42.547395+0000 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:42.548 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.547+0000 7f3538d82640 1 Processor -- start 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.547+0000 7f3538d82640 1 -- start start 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.547+0000 7f3538d82640 10 monclient: init 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 10 monclient(hunting): picked mon.noname-a con 0x7f35340b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 10 monclient(hunting): picked mon.noname-b con 0x7f35340b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 10 monclient(hunting): picked mon.noname-c con 0x7f35340b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35340a96f0 con 0x7f35340b4e70 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35340ab0e0 con 0x7f35340b1b00 2024-03-25T18:23:42.549 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3534145f70 con 0x7f35340b44b0 2024-03-25T18:23:42.550 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:42.550 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.548+0000 7f3538d82640 10 monclient(hunting): authenticate will time out at 7864.652817s 2024-03-25T18:23:42.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.550+0000 7f3532d9d640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f35340b1b00 0x7f353414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:46872/0 (socket says 172.21.15.7:46872) 2024-03-25T18:23:42.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.550+0000 7f3532d9d640 1 -- 172.21.15.7:0/437435918 learned_addr learned my addr 172.21.15.7:0/437435918 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:42.564 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.563+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3204714092 0 0) 0x7f35340ab0e0 con 0x7f35340b1b00 2024-03-25T18:23:42.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.563+0000 7f353159a640 10 monclient(hunting): _negotiate 2024-03-25T18:23:42.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.563+0000 7f353159a640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:42.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.563+0000 7f353159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:42.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.563+0000 7f353159a640 10 monclient(hunting): my global_id is 4621 2024-03-25T18:23:42.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.563+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f351c003520 con 0x7f35340b1b00 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1400879354 0 0) 0x7f351c003520 con 0x7f35340b1b00 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 >> v1:172.21.15.7:6790/0 conn(0x7f35340b44b0 legacy=0x7f3534145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 >> v1:172.21.15.7:6789/0 conn(0x7f35340b4e70 legacy=0x7f353413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 10 monclient: _finish_hunting 0 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 1 monclient: found mon.noname-c 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35340a9b80 con 0x7f35340b1b00 2024-03-25T18:23:42.581 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 10 monclient: _finish_auth 0 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 10 monclient: _check_auth_tickets 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f352c001a10 con 0x7f35340b1b00 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f353159a640 10 monclient: dump: 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:42.582 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:42.583 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:42.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f3538d82640 5 monclient: authenticate success, global_id 4621 2024-03-25T18:23:42.583 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.580+0000 7f3538d82640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f352c001b70 con 0x7f35340b1b00 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f353159a640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f353159a640 1 -- 172.21.15.7:0/437435918 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f352c001f80 con 0x7f35340b1b00 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f353159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f353159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f353159a640 10 monclient: dump: 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:42.591 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.592 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:42.592 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:42.592 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:42.592 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:42.592 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:42.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f3538d82640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:42.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f3538d82640 4 set_mon_vals no callback set 2024-03-25T18:23:42.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f3538d82640 10 monclient: shutdown 2024-03-25T18:23:42.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f3538d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:42.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.590+0000 7f3538d82640 1 -- 172.21.15.7:0/437435918 >> v1:172.21.15.183:6789/0 conn(0x7f35340b1b00 legacy=0x7f353414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.591+0000 7f3538d82640 1 -- 172.21.15.7:0/437435918 shutdown_connections 2024-03-25T18:23:42.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.591+0000 7f3538d82640 1 -- 172.21.15.7:0/437435918 >> 172.21.15.7:0/437435918 conn(0x7f35340617a0 msgr2=0x7f3534061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.591+0000 7f3538d82640 1 -- 172.21.15.7:0/437435918 shutdown_connections 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.591+0000 7f3538d82640 1 -- 172.21.15.7:0/437435918 wait complete. 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 10 monclient: build_initial_monmap 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 10 monclient: monmap: 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 1 librados: starting msgr at 2024-03-25T18:23:42.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 1 librados: starting objecter 2024-03-25T18:23:42.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 1 Processor -- start 2024-03-25T18:23:42.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 1 -- start start 2024-03-25T18:23:42.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 1 librados: setting wanted keys 2024-03-25T18:23:42.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 1 librados: calling monclient init 2024-03-25T18:23:42.595 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.592+0000 7f3538d82640 10 monclient: init 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 10 monclient(hunting): picked mon.noname-b con 0x7f35340b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 10 monclient(hunting): picked mon.noname-a con 0x7f35340b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 10 monclient(hunting): picked mon.noname-c con 0x7f35340b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35341d4540 con 0x7f35340b44b0 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3534142730 con 0x7f35340b1b00 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3534140160 con 0x7f35340b4e70 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.593+0000 7f3538d82640 10 monclient(hunting): authenticate will time out at 7864.698338s 2024-03-25T18:23:42.596 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.594+0000 7f3532d9d640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f35340b4e70 0x7f35341d50e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:55768/0 (socket says 172.21.15.7:55768) 2024-03-25T18:23:42.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.594+0000 7f3532d9d640 1 -- 172.21.15.7:0/2244010789 learned_addr learned my addr 172.21.15.7:0/2244010789 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:42.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.594+0000 7f3522ffd640 10 client.?.objecter ms_handle_connect 0x7f35340b4e70 2024-03-25T18:23:42.597 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.594+0000 7f3522ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:42.599 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.598+0000 7f3522ffd640 10 client.?.objecter ms_handle_connect 0x7f35340b1b00 2024-03-25T18:23:42.599 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.598+0000 7f3522ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:42.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.655+0000 7f3522ffd640 10 client.?.objecter ms_handle_connect 0x7f35340b44b0 2024-03-25T18:23:42.656 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.655+0000 7f3522ffd640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.673+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2482874659 0 0) 0x7f35341d4540 con 0x7f35340b44b0 2024-03-25T18:23:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.673+0000 7f3522ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.673+0000 7f3522ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.673+0000 7f3522ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.673+0000 7f3522ffd640 10 monclient(hunting): my global_id is 4623 2024-03-25T18:23:42.675 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.673+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3514004eb0 con 0x7f35340b44b0 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.699+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2083471329 0 0) 0x7f3514004eb0 con 0x7f35340b44b0 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.699+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f35341d4540 con 0x7f35340b44b0 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.700+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3528001930 con 0x7f35340b44b0 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.700+0000 7f3522ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.700+0000 7f3522ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.700+0000 7f3522ffd640 10 monclient(hunting): dump: 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:42.701 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:42.702 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:42.702 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:42.702 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:42.702 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:42.702 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:42.702 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:42.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.700+0000 7f3522ffd640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:42.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.716+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1504219364 0 0) 0x7f35341d4540 con 0x7f35340b44b0 2024-03-25T18:23:42.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 >> v1:172.21.15.7:6790/0 conn(0x7f35340b4e70 legacy=0x7f35341d50e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 >> v1:172.21.15.183:6789/0 conn(0x7f35340b1b00 legacy=0x7f3534140cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:42.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:42.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 1 monclient: found mon.a 2024-03-25T18:23:42.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:42.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35341402f0 con 0x7f35340b44b0 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 10 monclient: _finish_auth 0 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3522ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 5 monclient: authenticate success, global_id 4623 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 10 monclient: _renew_subs 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35341413e0 con 0x7f35340b44b0 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 10 client.4623.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 10 monclient: _renew_subs 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f35341e36e0 con 0x7f35340b44b0 2024-03-25T18:23:42.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.717+0000 7f3538d82640 1 librados: init done 2024-03-25T18:23:42.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.718+0000 7f3538d82640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:42.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.718+0000 7f3538d82640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:42.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.718+0000 7f3538d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:42.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:42.718+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35341d46f0 con 0x7f35340b44b0 2024-03-25T18:23:43.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.022+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3528003610 con 0x7f35340b44b0 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.022+0000 7f3522ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.022+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3528003840 con 0x7f35340b44b0 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.022+0000 7f3522ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.022+0000 7f3522ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.022+0000 7f3522ffd640 10 monclient: dump: 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:43.024 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:43.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.022+0000 7f3530d99640 4 set_mon_vals no callback set 2024-03-25T18:23:43.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.023+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f3528003a60 con 0x7f35340b44b0 2024-03-25T18:23:43.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.024+0000 7f353259c640 10 monclient: get_auth_request con 0x7f351406dd90 auth_method 0 2024-03-25T18:23:43.025 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.024+0000 7f3522ffd640 10 client.4623.objecter ms_handle_connect 0x7f351406dd90 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.029+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f3528053460 con 0x7f35340b44b0 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.029+0000 7f3522ffd640 10 client.4623.objecter ms_dispatch 0x7f3534000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.029+0000 7f3522ffd640 3 client.4623.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.030+0000 7f3522ffd640 3 client.4623.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.030+0000 7f3522ffd640 20 client.4623.objecter dump_active .. 0 homeless 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.030+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f352804ef70 con 0x7f35340b44b0 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.030+0000 7f3522ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:43.031 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.030+0000 7f3522ffd640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:43.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.286+0000 7f3538d82640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.286+0000 7f3538d82640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.286+0000 7f3538d82640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:43.287 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.286+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f353404f730 con 0x7f35340b44b0 2024-03-25T18:23:43.339 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.338+0000 7f3522ffd640 1 -- 172.21.15.7:0/2244010789 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f3528050ea0 con 0x7f35340b44b0 2024-03-25T18:23:43.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.338+0000 7f3522ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:43.340 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.338+0000 7f3522ffd640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:43.340 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:43.340 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 10 librados: watch_flush enter 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 10 librados: watch_flush exit 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 20 client.4623.objecter shutdown clearing up homeless session... 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 10 client.4623.objecter successfully canceled tick 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 >> v1:172.21.15.7:6800/1717564767 conn(0x7f351406dd90 legacy=0x7f3514070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 10 monclient: shutdown 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:43.343 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 >> v1:172.21.15.7:6789/0 conn(0x7f35340b44b0 legacy=0x7f35341d49d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:43.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 shutdown_connections 2024-03-25T18:23:43.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 >> 172.21.15.7:0/2244010789 conn(0x7f35340617a0 msgr2=0x7f35340b6590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:43.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 shutdown_connections 2024-03-25T18:23:43.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.341+0000 7f3538d82640 1 -- 172.21.15.7:0/2244010789 wait complete. 2024-03-25T18:23:43.344 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:43.342+0000 7f3538d82640 1 librados: shutdown 2024-03-25T18:23:43.354 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:50.160 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.281+0000 7fbea29cf640 10 monclient: get_monmap_and_config 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.281+0000 7fbea29cf640 10 monclient: build_initial_monmap 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.282+0000 7fbea29cf640 10 monclient: monmap: 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:50.282997+0000 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:50.282997+0000 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:50.283 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:50.284 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.283+0000 7fbea29cf640 1 Processor -- start 2024-03-25T18:23:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.283+0000 7fbea29cf640 1 -- start start 2024-03-25T18:23:50.284 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.283+0000 7fbea29cf640 10 monclient: init 2024-03-25T18:23:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:23:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 10 monclient(hunting): picked mon.noname-b con 0x7fbe9c0b4e70 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbe9c0b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9c0b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:50.285 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0a96f0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c0ab0e0 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c145f70 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:50.286 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.284+0000 7fbea29cf640 10 monclient(hunting): authenticate will time out at 7872.388776s 2024-03-25T18:23:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.306+0000 7fbe9b7fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbe9c0b44b0 0x7fbe9c145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:54736/0 (socket says 172.21.15.7:54736) 2024-03-25T18:23:50.308 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.306+0000 7fbe9b7fe640 1 -- 172.21.15.7:0/2973104088 learned_addr learned my addr 172.21.15.7:0/2973104088 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:50.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.326+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 874953093 0 0) 0x7fbe9c0ab0e0 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.326+0000 7fbe9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.326+0000 7fbe9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.327+0000 7fbe9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.327+0000 7fbe9affd640 10 monclient(hunting): my global_id is 4627 2024-03-25T18:23:50.328 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.327+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe84003520 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.329+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1445947779 0 0) 0x7fbe9c0a96f0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.329+0000 7fbe9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.329+0000 7fbe9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.329+0000 7fbe9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.330 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.329+0000 7fbe9affd640 10 monclient(hunting): my global_id is 4626 2024-03-25T18:23:50.331 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.329+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9c0ab0e0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.332 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.331+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602812612 0 0) 0x7fbe9c145f70 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.332 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.331+0000 7fbe9affd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.332 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.331+0000 7fbe9affd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.332 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.331+0000 7fbe9affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.332 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.331+0000 7fbe9affd640 10 monclient(hunting): my global_id is 4655 2024-03-25T18:23:50.332 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.331+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9c0a96f0 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.352 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2680433417 0 0) 0x7fbe84003520 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 >> v1:172.21.15.7:6790/0 conn(0x7fbe9c0b4e70 legacy=0x7fbe9c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 >> v1:172.21.15.7:6789/0 conn(0x7fbe9c0b44b0 legacy=0x7fbe9c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 1 monclient: found mon.noname-c 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c0a9b80 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 10 monclient: _finish_auth 0 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbe94001a10 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.353 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbe9affd640 10 monclient: dump: 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:50.354 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbea29cf640 5 monclient: authenticate success, global_id 4627 2024-03-25T18:23:50.355 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.351+0000 7fbea29cf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe94001b70 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbe9affd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbe9affd640 1 -- 172.21.15.7:0/2973104088 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbe94001f80 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbea29cf640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbea29cf640 4 set_mon_vals no callback set 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbe9affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbe9affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.376+0000 7fbe9affd640 10 monclient: dump: 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:50.378 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.379 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.379 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.379 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:50.379 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:50.379 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:50.379 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:50.379 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.377+0000 7fbea29cf640 10 monclient: shutdown 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.377+0000 7fbea29cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.377+0000 7fbea29cf640 1 -- 172.21.15.7:0/2973104088 >> v1:172.21.15.183:6789/0 conn(0x7fbe9c0b1b00 legacy=0x7fbe9c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.377+0000 7fbea29cf640 1 -- 172.21.15.7:0/2973104088 shutdown_connections 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.377+0000 7fbea29cf640 1 -- 172.21.15.7:0/2973104088 >> 172.21.15.7:0/2973104088 conn(0x7fbe9c0617a0 msgr2=0x7fbe9c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.377+0000 7fbea29cf640 1 -- 172.21.15.7:0/2973104088 shutdown_connections 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.377+0000 7fbea29cf640 1 -- 172.21.15.7:0/2973104088 wait complete. 2024-03-25T18:23:50.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 10 monclient: build_initial_monmap 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 10 monclient: monmap: 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 1 librados: starting msgr at 2024-03-25T18:23:50.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 1 librados: starting objecter 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 1 Processor -- start 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 1 -- start start 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 1 librados: setting wanted keys 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 1 librados: calling monclient init 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.378+0000 7fbea29cf640 10 monclient: init 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 10 monclient(hunting): picked mon.noname-b con 0x7fbe9c0b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 10 monclient(hunting): picked mon.noname-c con 0x7fbe9c0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 10 monclient(hunting): picked mon.noname-a con 0x7fbe9c0b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c143730 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c142010 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe9c144fa0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:50.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.379+0000 7fbea29cf640 10 monclient(hunting): authenticate will time out at 7872.484474s 2024-03-25T18:23:50.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.383+0000 7fbe9b7fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fbe9c0b44b0 0x7fbe9c1d77e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:60342/0 (socket says 172.21.15.7:60342) 2024-03-25T18:23:50.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.383+0000 7fbe9b7fe640 1 -- 172.21.15.7:0/4291811760 learned_addr learned my addr 172.21.15.7:0/4291811760 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:50.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.385+0000 7fbe98ff9640 10 client.?.objecter ms_handle_connect 0x7fbe9c0b4e70 2024-03-25T18:23:50.386 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.385+0000 7fbe98ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.388+0000 7fbe98ff9640 10 client.?.objecter ms_handle_connect 0x7fbe9c0b44b0 2024-03-25T18:23:50.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.388+0000 7fbe98ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.391+0000 7fbe98ff9640 10 client.?.objecter ms_handle_connect 0x7fbe9c0b1b00 2024-03-25T18:23:50.392 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.391+0000 7fbe98ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:50.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104446470 0 0) 0x7fbe9c144fa0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): my global_id is 4658 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe88004eb0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279872566 0 0) 0x7fbe9c143730 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 10 monclient(hunting): my global_id is 4629 2024-03-25T18:23:50.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.397+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9c144fa0 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.400+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586215809 0 0) 0x7fbe9c142010 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.400+0000 7fbe98ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:50.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.400+0000 7fbe98ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:50.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.400+0000 7fbe98ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:50.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.400+0000 7fbe98ff9640 10 monclient(hunting): my global_id is 4630 2024-03-25T18:23:50.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.400+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe9c143730 con 0x7fbe9c0b1b00 2024-03-25T18:23:50.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.404+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 915200899 0 0) 0x7fbe9c144fa0 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.405 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.404+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe9c142010 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.406+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbe94001e80 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.406+0000 7fbe98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.406+0000 7fbe98ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.406+0000 7fbe98ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.407 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.408 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:50.408 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:50.408 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:50.408 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:50.408 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:50.408 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.406+0000 7fbe98ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:50.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.414+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4067751684 0 0) 0x7fbe88004eb0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.414+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe9c144fa0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.414+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbe900019b0 con 0x7fbe9c0b44b0 2024-03-25T18:23:50.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.414+0000 7fbe98ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.414+0000 7fbe98ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:23:50.415 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.414+0000 7fbe98ff9640 10 monclient(hunting): dump: 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:50.416 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4017301122 0 0) 0x7fbe9c142010 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 >> v1:172.21.15.7:6790/0 conn(0x7fbe9c0b44b0 legacy=0x7fbe9c1d77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 >> v1:172.21.15.183:6789/0 conn(0x7fbe9c0b1b00 legacy=0x7fbe9c141340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 1 monclient: found mon.a 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe9c145130 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:50.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbe98ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.415+0000 7fbea29cf640 5 monclient: authenticate success, global_id 4629 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 10 monclient: _renew_subs 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe9c1e3200 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 10 client.4629.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 10 monclient: _renew_subs 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe9c1e3570 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.416+0000 7fbea29cf640 1 librados: init done 2024-03-25T18:23:50.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.417+0000 7fbea29cf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:50.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.417+0000 7fbea29cf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:50.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.417+0000 7fbea29cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:50.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.417+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe9c142700 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.737+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe94001fe0 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.737+0000 7fbe98ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.737+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbe94003c20 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.737+0000 7fbe98ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.737+0000 7fbe98ff9640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.737+0000 7fbe98ff9640 10 monclient: dump: 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:50.739 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:50.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.737+0000 7fbe9a7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:50.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.738+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fbe94019bf0 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.738+0000 7fbe9bfff640 10 monclient: get_auth_request con 0x7fbe8806dec0 auth_method 0 2024-03-25T18:23:50.740 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.739+0000 7fbe98ff9640 10 client.4629.objecter ms_handle_connect 0x7fbe8806dec0 2024-03-25T18:23:50.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7fbe940538c0 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 10 client.4629.objecter ms_dispatch 0x7fbe9c000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:50.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 3 client.4629.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:50.749 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 3 client.4629.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:50.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 20 client.4629.objecter dump_active .. 0 homeless 2024-03-25T18:23:50.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe9404f200 con 0x7fbe9c0b4e70 2024-03-25T18:23:50.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:50.750 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.748+0000 7fbe98ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.998+0000 7fbea29cf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.998+0000 7fbea29cf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.998+0000 7fbea29cf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:23:50.999 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:50.998+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe9c04f730 con 0x7fbe9c0b4e70 2024-03-25T18:23:51.080 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.078+0000 7fbe98ff9640 1 -- 172.21.15.7:0/4291811760 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7fbe94051300 con 0x7fbe9c0b4e70 2024-03-25T18:23:51.080 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.078+0000 7fbe98ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:51.080 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.078+0000 7fbe98ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:51.080 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:51.080 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:51.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.080+0000 7fbea29cf640 10 librados: watch_flush enter 2024-03-25T18:23:51.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.080+0000 7fbea29cf640 10 librados: watch_flush exit 2024-03-25T18:23:51.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.080+0000 7fbea29cf640 20 client.4629.objecter shutdown clearing up homeless session... 2024-03-25T18:23:51.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.080+0000 7fbea29cf640 10 client.4629.objecter successfully canceled tick 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.080+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 >> v1:172.21.15.7:6800/1717564767 conn(0x7fbe8806dec0 legacy=0x7fbe88070380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.080+0000 7fbea29cf640 10 monclient: shutdown 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.081+0000 7fbea29cf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.081+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 >> v1:172.21.15.7:6789/0 conn(0x7fbe9c0b4e70 legacy=0x7fbe9c1d7ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.081+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 shutdown_connections 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.081+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 >> 172.21.15.7:0/4291811760 conn(0x7fbe9c0617a0 msgr2=0x7fbe9c13ee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.081+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 shutdown_connections 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.081+0000 7fbea29cf640 1 -- 172.21.15.7:0/4291811760 wait complete. 2024-03-25T18:23:51.083 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:51.081+0000 7fbea29cf640 1 librados: shutdown 2024-03-25T18:23:51.093 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:23:54.253 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:00.890 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.315 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:00.951 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.356 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:00.992 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.361 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:00.998 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.368 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:00.994 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.375 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:01.012 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.376 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:01.002 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.400 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:01.037 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:54.411 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:01.037 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:23:57.899 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.018+0000 7f0f5698d640 10 monclient: get_monmap_and_config 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.018+0000 7f0f5698d640 10 monclient: build_initial_monmap 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.019+0000 7f0f5698d640 10 monclient: monmap: 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:23:58.020008+0000 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:23:58.020008+0000 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:23:58.020 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.019+0000 7f0f5698d640 1 Processor -- start 2024-03-25T18:23:58.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.019+0000 7f0f5698d640 1 -- start start 2024-03-25T18:23:58.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.019+0000 7f0f5698d640 10 monclient: init 2024-03-25T18:23:58.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:58.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:23:58.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 10 monclient(hunting): picked mon.noname-b con 0x7f0f500b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:58.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f500b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:58.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:58.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f500ab0e0 con 0x7f0f5013c480 2024-03-25T18:23:58.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f500ab3c0 con 0x7f0f500b4670 2024-03-25T18:23:58.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f500a9f30 con 0x7f0f500b4ff0 2024-03-25T18:23:58.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:58.023 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.020+0000 7f0f5698d640 10 monclient(hunting): authenticate will time out at 7880.125615s 2024-03-25T18:23:58.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.036+0000 7f0f54f2a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0f5013c480 0x7f0f5014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:43966/0 (socket says 172.21.15.7:43966) 2024-03-25T18:23:58.037 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.036+0000 7f0f54f2a640 1 -- 172.21.15.7:0/4266634148 learned_addr learned my addr 172.21.15.7:0/4266634148 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:58.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.051+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1070284507 0 0) 0x7f0f500ab3c0 con 0x7f0f500b4670 2024-03-25T18:23:58.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.052+0000 7f0f4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.052+0000 7f0f4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.052+0000 7f0f4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.052+0000 7f0f4effd640 10 monclient(hunting): my global_id is 4633 2024-03-25T18:23:58.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.052+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f38003520 con 0x7f0f500b4670 2024-03-25T18:23:58.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.057+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1289757769 0 0) 0x7f0f500ab0e0 con 0x7f0f5013c480 2024-03-25T18:23:58.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.057+0000 7f0f4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.057+0000 7f0f4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.057+0000 7f0f4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.057+0000 7f0f4effd640 10 monclient(hunting): my global_id is 4632 2024-03-25T18:23:58.059 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.057+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f500ab3c0 con 0x7f0f5013c480 2024-03-25T18:23:58.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.060+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3510859390 0 0) 0x7f0f500a9f30 con 0x7f0f500b4ff0 2024-03-25T18:23:58.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.060+0000 7f0f4effd640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.060+0000 7f0f4effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.060+0000 7f0f4effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.061+0000 7f0f4effd640 10 monclient(hunting): my global_id is 4661 2024-03-25T18:23:58.062 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.061+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f500ab0e0 con 0x7f0f500b4ff0 2024-03-25T18:23:58.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3966783659 0 0) 0x7f0f500ab3c0 con 0x7f0f5013c480 2024-03-25T18:23:58.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 >> v1:172.21.15.7:6790/0 conn(0x7f0f500b4ff0 legacy=0x7f0f5013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 >> v1:172.21.15.183:6789/0 conn(0x7f0f500b4670 legacy=0x7f0f501425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 10 monclient: _finish_hunting 0 2024-03-25T18:23:58.068 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 1 monclient: found mon.noname-a 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f500a9b70 con 0x7f0f5013c480 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 10 monclient: _finish_auth 0 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 10 monclient: _check_auth_tickets 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0f480019f0 con 0x7f0f5013c480 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f4effd640 10 monclient: dump: 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.069 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:58.070 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:58.070 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:58.070 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:58.070 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.070 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f5698d640 5 monclient: authenticate success, global_id 4632 2024-03-25T18:23:58.070 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.067+0000 7f0f5698d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.087+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f48001b50 con 0x7f0f5013c480 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.087+0000 7f0f4effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.087+0000 7f0f4effd640 1 -- 172.21.15.7:0/4266634148 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0f48001f60 con 0x7f0f5013c480 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.087+0000 7f0f4effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.087+0000 7f0f4effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.087+0000 7f0f4effd640 10 monclient: dump: 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.089 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:58.090 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:58.090 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:58.090 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:58.090 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.088+0000 7f0f5698d640 10 monclient: get_monmap_and_config success 2024-03-25T18:23:58.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.088+0000 7f0f5698d640 4 set_mon_vals no callback set 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.088+0000 7f0f5698d640 10 monclient: shutdown 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.088+0000 7f0f5698d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.088+0000 7f0f5698d640 1 -- 172.21.15.7:0/4266634148 >> v1:172.21.15.7:6789/0 conn(0x7f0f5013c480 legacy=0x7f0f5014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.089+0000 7f0f5698d640 1 -- 172.21.15.7:0/4266634148 shutdown_connections 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.089+0000 7f0f5698d640 1 -- 172.21.15.7:0/4266634148 >> 172.21.15.7:0/4266634148 conn(0x7f0f500617a0 msgr2=0x7f0f50061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.089+0000 7f0f5698d640 1 -- 172.21.15.7:0/4266634148 shutdown_connections 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.089+0000 7f0f5698d640 1 -- 172.21.15.7:0/4266634148 wait complete. 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 10 monclient: build_initial_monmap 2024-03-25T18:23:58.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 10 monclient: monmap: 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 1 librados: starting msgr at 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 1 librados: starting objecter 2024-03-25T18:23:58.092 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 1 Processor -- start 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 1 -- start start 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 1 librados: setting wanted keys 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 1 librados: calling monclient init 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.090+0000 7f0f5698d640 10 monclient: init 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 10 monclient: _reopen_session rank -1 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 10 monclient(hunting): picked mon.noname-b con 0x7f0f500b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 10 monclient(hunting): picked mon.noname-c con 0x7f0f500b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 10 monclient(hunting): picked mon.noname-a con 0x7f0f5013c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:23:58.093 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f50137ce0 con 0x7f0f5013c480 2024-03-25T18:23:58.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f50132b80 con 0x7f0f500b4670 2024-03-25T18:23:58.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f50137fc0 con 0x7f0f500b4ff0 2024-03-25T18:23:58.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 10 monclient(hunting): _renew_subs 2024-03-25T18:23:58.094 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.091+0000 7f0f5698d640 10 monclient(hunting): authenticate will time out at 7880.196164s 2024-03-25T18:23:58.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.095+0000 7f0f54f2a640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f0f5013c480 0x7f0f50132470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:44020/0 (socket says 172.21.15.7:44020) 2024-03-25T18:23:58.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.095+0000 7f0f54f2a640 1 -- 172.21.15.7:0/2288036025 learned_addr learned my addr 172.21.15.7:0/2288036025 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:23:58.102 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.101+0000 7f0f4cff9640 10 client.?.objecter ms_handle_connect 0x7f0f5013c480 2024-03-25T18:23:58.102 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.101+0000 7f0f4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.109+0000 7f0f4cff9640 10 client.?.objecter ms_handle_connect 0x7f0f500b4ff0 2024-03-25T18:23:58.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.109+0000 7f0f4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.110+0000 7f0f4cff9640 10 client.?.objecter ms_handle_connect 0x7f0f500b4670 2024-03-25T18:23:58.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.110+0000 7f0f4cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:23:58.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.121+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4216575101 0 0) 0x7f0f50137ce0 con 0x7f0f5013c480 2024-03-25T18:23:58.122 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.121+0000 7f0f4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.121+0000 7f0f4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.121+0000 7f0f4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.121+0000 7f0f4cff9640 10 monclient(hunting): my global_id is 4635 2024-03-25T18:23:58.123 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.121+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f3c004eb0 con 0x7f0f5013c480 2024-03-25T18:23:58.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.124+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 989533949 0 0) 0x7f0f50137fc0 con 0x7f0f500b4ff0 2024-03-25T18:23:58.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.124+0000 7f0f4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.124+0000 7f0f4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.125 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.124+0000 7f0f4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.124+0000 7f0f4cff9640 10 monclient(hunting): my global_id is 4664 2024-03-25T18:23:58.126 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.124+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f50137ce0 con 0x7f0f500b4ff0 2024-03-25T18:23:58.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.126+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3987777789 0 0) 0x7f0f50132b80 con 0x7f0f500b4670 2024-03-25T18:23:58.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.126+0000 7f0f4cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:23:58.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.126+0000 7f0f4cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:23:58.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.126+0000 7f0f4cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:23:58.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.126+0000 7f0f4cff9640 10 monclient(hunting): my global_id is 4636 2024-03-25T18:23:58.127 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.126+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0f50137fc0 con 0x7f0f500b4670 2024-03-25T18:23:58.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.140+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2442862142 0 0) 0x7f0f50137ce0 con 0x7f0f500b4ff0 2024-03-25T18:23:58.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.140+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f50132b80 con 0x7f0f500b4ff0 2024-03-25T18:23:58.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.140+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0f44001950 con 0x7f0f500b4ff0 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.140+0000 7f0f4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.140+0000 7f0f4cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.140+0000 7f0f4cff9640 10 monclient(hunting): dump: 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.140+0000 7f0f4cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:23:58.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3344424916 0 0) 0x7f0f3c004eb0 con 0x7f0f5013c480 2024-03-25T18:23:58.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f50137ce0 con 0x7f0f5013c480 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0f48001f60 con 0x7f0f5013c480 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 10 monclient(hunting): dump: 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:58.146 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1563221622 0 0) 0x7f0f50137fc0 con 0x7f0f500b4670 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0f3c004eb0 con 0x7f0f500b4670 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0f40001830 con 0x7f0f500b4670 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.144+0000 7f0f4cff9640 10 monclient(hunting): dump: 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:58.147 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:58.148 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4143343004 0 0) 0x7f0f3c004eb0 con 0x7f0f500b4670 2024-03-25T18:23:58.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 >> v1:172.21.15.7:6790/0 conn(0x7f0f500b4ff0 legacy=0x7f0f50131d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 >> v1:172.21.15.7:6789/0 conn(0x7f0f5013c480 legacy=0x7f0f50132470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 1 monclient: found mon.b 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f501382a0 con 0x7f0f500b4670 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 10 monclient: _finish_auth 0 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f4cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f5698d640 5 monclient: authenticate success, global_id 4636 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.158+0000 7f0f5698d640 10 monclient: _renew_subs 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.159+0000 7f0f5698d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.159+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5013de70 con 0x7f0f500b4670 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.159+0000 7f0f5698d640 10 client.4636.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.159+0000 7f0f5698d640 10 monclient: _renew_subs 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.159+0000 7f0f5698d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.159+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5013b9c0 con 0x7f0f500b4670 2024-03-25T18:23:58.161 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.159+0000 7f0f5698d640 1 librados: init done 2024-03-25T18:23:58.162 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.160+0000 7f0f5698d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.162 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.160+0000 7f0f5698d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.162 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.160+0000 7f0f5698d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:58.162 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.160+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5006dbd0 con 0x7f0f500b4670 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.455+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0f40001d40 con 0x7f0f500b4670 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.455+0000 7f0f4cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.455+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f0f400037b0 con 0x7f0f500b4670 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.455+0000 7f0f4cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.455+0000 7f0f4cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.455+0000 7f0f4cff9640 10 monclient: dump: 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:23:58.457 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:23:58.458 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:23:58.458 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:23:58.458 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:23:58.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.455+0000 7f0f4e7fc640 4 set_mon_vals no callback set 2024-03-25T18:23:58.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.456+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f0f40019800 con 0x7f0f500b4670 2024-03-25T18:23:58.458 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.457+0000 7f0f4f7fe640 10 monclient: get_auth_request con 0x7f0f3c06e160 auth_method 0 2024-03-25T18:23:58.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.457+0000 7f0f4cff9640 10 client.4636.objecter ms_handle_connect 0x7f0f3c06e160 2024-03-25T18:23:58.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f0f40053580 con 0x7f0f500b4670 2024-03-25T18:23:58.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 10 client.4636.objecter ms_dispatch 0x7f0f50000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:23:58.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 3 client.4636.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:23:58.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 3 client.4636.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:23:58.463 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 20 client.4636.objecter dump_active .. 0 homeless 2024-03-25T18:23:58.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0f4004eec0 con 0x7f0f500b4670 2024-03-25T18:23:58.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:23:58.464 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.462+0000 7f0f4cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:23:58.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.686+0000 7f0f5698d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:58.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.686+0000 7f0f5698d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:58.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.686+0000 7f0f5698d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:23:58.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.686+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0f500a9450 con 0x7f0f500b4670 2024-03-25T18:23:58.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.723+0000 7f0f4cff9640 1 -- 172.21.15.7:0/2288036025 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f0f40050fc0 con 0x7f0f500b4670 2024-03-25T18:23:58.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.723+0000 7f0f4cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:23:58.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.723+0000 7f0f4cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:23:58.725 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:23:58.725 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:23:58.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.725+0000 7f0f5698d640 10 librados: watch_flush enter 2024-03-25T18:23:58.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.725+0000 7f0f5698d640 10 librados: watch_flush exit 2024-03-25T18:23:58.727 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 20 client.4636.objecter shutdown clearing up homeless session... 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 10 client.4636.objecter successfully canceled tick 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 >> v1:172.21.15.7:6800/1717564767 conn(0x7f0f3c06e160 legacy=0x7f0f3c070620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 10 monclient: shutdown 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 >> v1:172.21.15.183:6789/0 conn(0x7f0f500b4670 legacy=0x7f0f5013ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 shutdown_connections 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 >> 172.21.15.7:0/2288036025 conn(0x7f0f500617a0 msgr2=0x7f0f50141ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 shutdown_connections 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 1 -- 172.21.15.7:0/2288036025 wait complete. 2024-03-25T18:23:58.728 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:23:58.726+0000 7f0f5698d640 1 librados: shutdown 2024-03-25T18:23:58.739 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:05.545 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:05.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.642+0000 7f99da33a640 10 monclient: get_monmap_and_config 2024-03-25T18:24:05.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.642+0000 7f99da33a640 10 monclient: build_initial_monmap 2024-03-25T18:24:05.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.642+0000 7f99da33a640 10 monclient: monmap: 2024-03-25T18:24:05.643 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:05.643 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:05.643482+0000 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:05.643482+0000 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:05.644 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:05.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 1 Processor -- start 2024-03-25T18:24:05.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 1 -- start start 2024-03-25T18:24:05.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient: init 2024-03-25T18:24:05.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:05.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:05.645 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient(hunting): picked mon.noname-a con 0x7f99d40b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient(hunting): picked mon.noname-c con 0x7f99d40b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient(hunting): picked mon.noname-b con 0x7f99d40b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d40a96f0 con 0x7f99d40b4e70 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d40ab0e0 con 0x7f99d40b44b0 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4145f70 con 0x7f99d40b1b00 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.643+0000 7f99da33a640 10 monclient(hunting): authenticate will time out at 7887.748608s 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.644+0000 7f99d37fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f99d40b44b0 0x7f99d4145860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:37022/0 (socket says 172.21.15.7:37022) 2024-03-25T18:24:05.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.644+0000 7f99d37fe640 1 -- 172.21.15.7:0/1044591816 learned_addr learned my addr 172.21.15.7:0/1044591816 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:05.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.661+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5017681 0 0) 0x7f99d40ab0e0 con 0x7f99d40b44b0 2024-03-25T18:24:05.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.662+0000 7f99d2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:05.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.662+0000 7f99d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:05.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.662+0000 7f99d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:05.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.662+0000 7f99d2ffd640 10 monclient(hunting): my global_id is 4639 2024-03-25T18:24:05.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.662+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99bc003520 con 0x7f99d40b44b0 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1096707719 0 0) 0x7f99bc003520 con 0x7f99d40b44b0 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 >> v1:172.21.15.7:6790/0 conn(0x7f99d40b1b00 legacy=0x7f99d414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 >> v1:172.21.15.7:6789/0 conn(0x7f99d40b4e70 legacy=0x7f99d413f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 1 monclient: found mon.noname-c 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d40a9b80 con 0x7f99d40b44b0 2024-03-25T18:24:05.673 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:05.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:05.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:05.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99da33a640 5 monclient: authenticate success, global_id 4639 2024-03-25T18:24:05.674 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.672+0000 7f99da33a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:05.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.677+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f99c8001690 con 0x7f99d40b44b0 2024-03-25T18:24:05.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.677+0000 7f99d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:05.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.677+0000 7f99d2ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:05.678 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.677+0000 7f99d2ffd640 10 monclient: dump: 2024-03-25T18:24:05.678 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:05.678 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:05.678 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:05.679 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.677+0000 7f99da33a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99c8001960 con 0x7f99d40b44b0 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99d2ffd640 1 -- 172.21.15.7:0/1044591816 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f99c8001cf0 con 0x7f99d40b44b0 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99d2ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99d2ffd640 10 monclient: dump: 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:05.685 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99da33a640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:05.686 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.684+0000 7f99da33a640 4 set_mon_vals no callback set 2024-03-25T18:24:05.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.685+0000 7f99da33a640 10 monclient: shutdown 2024-03-25T18:24:05.687 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.685+0000 7f99da33a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.685+0000 7f99da33a640 1 -- 172.21.15.7:0/1044591816 >> v1:172.21.15.183:6789/0 conn(0x7f99d40b44b0 legacy=0x7f99d4145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.685+0000 7f99da33a640 1 -- 172.21.15.7:0/1044591816 shutdown_connections 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.685+0000 7f99da33a640 1 -- 172.21.15.7:0/1044591816 >> 172.21.15.7:0/1044591816 conn(0x7f99d40617a0 msgr2=0x7f99d4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.685+0000 7f99da33a640 1 -- 172.21.15.7:0/1044591816 shutdown_connections 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.685+0000 7f99da33a640 1 -- 172.21.15.7:0/1044591816 wait complete. 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 10 monclient: build_initial_monmap 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 10 monclient: monmap: 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:05.688 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 1 librados: starting msgr at 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 1 librados: starting objecter 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 1 Processor -- start 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 1 -- start start 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 1 librados: setting wanted keys 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 1 librados: calling monclient init 2024-03-25T18:24:05.689 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.686+0000 7f99da33a640 10 monclient: init 2024-03-25T18:24:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 10 monclient(hunting): picked mon.noname-c con 0x7f99d40b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 10 monclient(hunting): picked mon.noname-a con 0x7f99d40b44b0 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:05.690 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 10 monclient(hunting): picked mon.noname-b con 0x7f99d40b4e70 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4136e10 con 0x7f99d40b44b0 2024-03-25T18:24:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d4137b40 con 0x7f99d40b4e70 2024-03-25T18:24:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99d41368f0 con 0x7f99d40b1b00 2024-03-25T18:24:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:05.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.687+0000 7f99da33a640 10 monclient(hunting): authenticate will time out at 7887.792385s 2024-03-25T18:24:05.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.693+0000 7f99d88d7640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f99d40b4e70 0x7f99d41e2e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:37048/0 (socket says 172.21.15.7:37048) 2024-03-25T18:24:05.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.693+0000 7f99d88d7640 1 -- 172.21.15.7:0/766426254 learned_addr learned my addr 172.21.15.7:0/766426254 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:05.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.694+0000 7f99d0ff9640 10 client.?.objecter ms_handle_connect 0x7f99d40b4e70 2024-03-25T18:24:05.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.694+0000 7f99d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:05.697 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.697+0000 7f99d0ff9640 10 client.?.objecter ms_handle_connect 0x7f99d40b1b00 2024-03-25T18:24:05.698 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.697+0000 7f99d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:05.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.707+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2934659851 0 0) 0x7f99d4137b40 con 0x7f99d40b4e70 2024-03-25T18:24:05.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.707+0000 7f99d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:05.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.707+0000 7f99d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:05.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.707+0000 7f99d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:05.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.707+0000 7f99d0ff9640 10 monclient(hunting): my global_id is 4642 2024-03-25T18:24:05.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.707+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c0004eb0 con 0x7f99d40b4e70 2024-03-25T18:24:05.709 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.708+0000 7f99d0ff9640 10 client.?.objecter ms_handle_connect 0x7f99d40b44b0 2024-03-25T18:24:05.710 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.708+0000 7f99d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:05.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.721+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748845709 0 0) 0x7f99d41368f0 con 0x7f99d40b1b00 2024-03-25T18:24:05.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.721+0000 7f99d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:05.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.721+0000 7f99d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:05.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.721+0000 7f99d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:05.722 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.721+0000 7f99d0ff9640 10 monclient(hunting): my global_id is 4670 2024-03-25T18:24:05.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.721+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99d4137b40 con 0x7f99d40b1b00 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.722+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1127813122 0 0) 0x7f99c0004eb0 con 0x7f99d40b4e70 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.722+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f99d41368f0 con 0x7f99d40b4e70 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.722+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f99cc001d30 con 0x7f99d40b4e70 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.722+0000 7f99d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.722+0000 7f99d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.722+0000 7f99d0ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:05.724 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:05.725 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:05.725 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:05.725 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:05.725 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.722+0000 7f99d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:05.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.725+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3668014216 0 0) 0x7f99d4136e10 con 0x7f99d40b44b0 2024-03-25T18:24:05.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.725+0000 7f99d0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:05.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.725+0000 7f99d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:05.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.725+0000 7f99d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:05.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.725+0000 7f99d0ff9640 10 monclient(hunting): my global_id is 4638 2024-03-25T18:24:05.726 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.725+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f99c0004eb0 con 0x7f99d40b44b0 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3496493110 0 0) 0x7f99d41368f0 con 0x7f99d40b4e70 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 >> v1:172.21.15.7:6790/0 conn(0x7f99d40b1b00 legacy=0x7f99d4133e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 >> v1:172.21.15.7:6789/0 conn(0x7f99d40b44b0 legacy=0x7f99d41d7650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 1 monclient: found mon.b 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99d4134de0 con 0x7f99d40b4e70 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 5 monclient: authenticate success, global_id 4642 2024-03-25T18:24:05.736 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 10 monclient: _renew_subs 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99d4139640 con 0x7f99d40b4e70 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 10 client.4642.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 10 monclient: _renew_subs 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f99d4135c10 con 0x7f99d40b4e70 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.734+0000 7f99da33a640 1 librados: init done 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.735+0000 7f99da33a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:05.737 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.735+0000 7f99da33a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:05.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.735+0000 7f99da33a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:05.738 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:05.735+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99d4136740 con 0x7f99d40b4e70 2024-03-25T18:24:06.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.007+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f99cc003990 con 0x7f99d40b4e70 2024-03-25T18:24:06.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.007+0000 7f99d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:06.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.008+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f99cc003ca0 con 0x7f99d40b4e70 2024-03-25T18:24:06.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.007+0000 7f99d27fc640 4 set_mon_vals no callback set 2024-03-25T18:24:06.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.008+0000 7f99d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:06.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.008+0000 7f99d0ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.008+0000 7f99d0ff9640 10 monclient: dump: 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:06.010 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:06.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.008+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f99cc003ec0 con 0x7f99d40b4e70 2024-03-25T18:24:06.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.009+0000 7f99d3fff640 10 monclient: get_auth_request con 0x7f99c006de00 auth_method 0 2024-03-25T18:24:06.011 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.009+0000 7f99d0ff9640 10 client.4642.objecter ms_handle_connect 0x7f99c006de00 2024-03-25T18:24:06.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f99cc054aa0 con 0x7f99d40b4e70 2024-03-25T18:24:06.015 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 10 client.4642.objecter ms_dispatch 0x7f99d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:06.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 3 client.4642.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:06.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 3 client.4642.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:06.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 20 client.4642.objecter dump_active .. 0 homeless 2024-03-25T18:24:06.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f99cc0505b0 con 0x7f99d40b4e70 2024-03-25T18:24:06.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:06.016 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.014+0000 7f99d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:06.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.266+0000 7f99da33a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.266+0000 7f99da33a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.266+0000 7f99da33a640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:06.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.266+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f99d4136a80 con 0x7f99d40b4e70 2024-03-25T18:24:06.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.296+0000 7f99d0ff9640 1 -- 172.21.15.7:0/766426254 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f99cc0524e0 con 0x7f99d40b4e70 2024-03-25T18:24:06.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.296+0000 7f99d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:06.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.296+0000 7f99d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:06.298 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:06.298 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 10 librados: watch_flush enter 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 10 librados: watch_flush exit 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 20 client.4642.objecter shutdown clearing up homeless session... 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 10 client.4642.objecter successfully canceled tick 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 >> v1:172.21.15.7:6800/1717564767 conn(0x7f99c006de00 legacy=0x7f99c00702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 10 monclient: shutdown 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 >> v1:172.21.15.183:6789/0 conn(0x7f99d40b4e70 legacy=0x7f99d41e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 shutdown_connections 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 >> 172.21.15.7:0/766426254 conn(0x7f99d40617a0 msgr2=0x7f99d40b59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:06.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 shutdown_connections 2024-03-25T18:24:06.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 1 -- 172.21.15.7:0/766426254 wait complete. 2024-03-25T18:24:06.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:06.299+0000 7f99da33a640 1 librados: shutdown 2024-03-25T18:24:06.312 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:13.119 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.233+0000 7f5ffd27d640 10 monclient: get_monmap_and_config 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.233+0000 7f5ffd27d640 10 monclient: build_initial_monmap 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.234+0000 7f5ffd27d640 10 monclient: monmap: 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:13.235023+0000 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:13.235023+0000 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:13.235 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.234+0000 7f5ffd27d640 1 Processor -- start 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 1 -- start start 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient: init 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ff80b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ff80b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ff813c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff80ab0e0 con 0x7f5ff80b4670 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff80ab3c0 con 0x7f5ff813c480 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff80a9f30 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:13.237 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.235+0000 7f5ffd27d640 10 monclient(hunting): authenticate will time out at 7895.340407s 2024-03-25T18:24:13.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.247+0000 7f5feffff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5ff80b4670 0x7f5ff81425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55108/0 (socket says 172.21.15.7:55108) 2024-03-25T18:24:13.249 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.248+0000 7f5feffff640 1 -- 172.21.15.7:0/4103279908 learned_addr learned my addr 172.21.15.7:0/4103279908 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:13.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.265+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2731177237 0 0) 0x7f5ff80ab0e0 con 0x7f5ff80b4670 2024-03-25T18:24:13.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.265+0000 7f5ff67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.265+0000 7f5ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.265+0000 7f5ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.265+0000 7f5ff67fc640 10 monclient(hunting): my global_id is 4641 2024-03-25T18:24:13.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.265+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fe4003520 con 0x7f5ff80b4670 2024-03-25T18:24:13.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.267+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 181947954 0 0) 0x7f5ff80ab3c0 con 0x7f5ff813c480 2024-03-25T18:24:13.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.267+0000 7f5ff67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.267+0000 7f5ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.267+0000 7f5ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.269 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.267+0000 7f5ff67fc640 10 monclient(hunting): my global_id is 4645 2024-03-25T18:24:13.269 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.267+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff80ab0e0 con 0x7f5ff813c480 2024-03-25T18:24:13.269 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.268+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3476325013 0 0) 0x7f5ff80a9f30 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.270 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.268+0000 7f5ff67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.270 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.268+0000 7f5ff67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.270 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.268+0000 7f5ff67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.270 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.268+0000 7f5ff67fc640 10 monclient(hunting): my global_id is 4673 2024-03-25T18:24:13.270 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.268+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff80ab3c0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 628734687 0 0) 0x7f5ff80ab0e0 con 0x7f5ff813c480 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 >> v1:172.21.15.7:6790/0 conn(0x7f5ff80b4ff0 legacy=0x7f5ff813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 >> v1:172.21.15.7:6789/0 conn(0x7f5ff80b4670 legacy=0x7f5ff81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 1 monclient: found mon.noname-c 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff80a9b70 con 0x7f5ff813c480 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5ff00019f0 con 0x7f5ff813c480 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.288 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ff67fc640 10 monclient: dump: 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ffd27d640 5 monclient: authenticate success, global_id 4645 2024-03-25T18:24:13.289 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.286+0000 7f5ffd27d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5ff0001b50 con 0x7f5ff813c480 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ff67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ff67fc640 1 -- 172.21.15.7:0/4103279908 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5ff0001fa0 con 0x7f5ff813c480 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ff67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ff67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ff67fc640 10 monclient: dump: 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:13.297 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:13.298 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:13.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ffd27d640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:13.298 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.295+0000 7f5ffd27d640 4 set_mon_vals no callback set 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.296+0000 7f5ffd27d640 10 monclient: shutdown 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.296+0000 7f5ffd27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.296+0000 7f5ffd27d640 1 -- 172.21.15.7:0/4103279908 >> v1:172.21.15.183:6789/0 conn(0x7f5ff813c480 legacy=0x7f5ff814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.296+0000 7f5ffd27d640 1 -- 172.21.15.7:0/4103279908 shutdown_connections 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.296+0000 7f5ffd27d640 1 -- 172.21.15.7:0/4103279908 >> 172.21.15.7:0/4103279908 conn(0x7f5ff80617a0 msgr2=0x7f5ff8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.296+0000 7f5ffd27d640 1 -- 172.21.15.7:0/4103279908 shutdown_connections 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.297+0000 7f5ffd27d640 1 -- 172.21.15.7:0/4103279908 wait complete. 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.297+0000 7f5ffd27d640 10 monclient: build_initial_monmap 2024-03-25T18:24:13.299 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.297+0000 7f5ffd27d640 10 monclient: monmap: 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.298+0000 7f5ffd27d640 1 librados: starting msgr at 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.298+0000 7f5ffd27d640 1 librados: starting objecter 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.298+0000 7f5ffd27d640 1 Processor -- start 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.298+0000 7f5ffd27d640 1 -- start start 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.298+0000 7f5ffd27d640 1 librados: setting wanted keys 2024-03-25T18:24:13.300 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.298+0000 7f5ffd27d640 1 librados: calling monclient init 2024-03-25T18:24:13.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.298+0000 7f5ffd27d640 10 monclient: init 2024-03-25T18:24:13.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:13.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:13.301 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 10 monclient(hunting): picked mon.noname-c con 0x7f5ff80b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:13.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 10 monclient(hunting): picked mon.noname-b con 0x7f5ff80b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:13.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 10 monclient(hunting): picked mon.noname-a con 0x7f5ff813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:13.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff8132bf0 con 0x7f5ff813c480 2024-03-25T18:24:13.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff8138100 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ff8143620 con 0x7f5ff80b4670 2024-03-25T18:24:13.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:13.302 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.299+0000 7f5ffd27d640 10 monclient(hunting): authenticate will time out at 7895.404005s 2024-03-25T18:24:13.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.303+0000 7f5ff77fe640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f5ff813c480 0x7f5ff81324e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:55158/0 (socket says 172.21.15.7:55158) 2024-03-25T18:24:13.304 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.303+0000 7f5ff77fe640 1 -- 172.21.15.7:0/3498439326 learned_addr learned my addr 172.21.15.7:0/3498439326 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:13.307 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.306+0000 7f5fef7fe640 10 client.?.objecter ms_handle_connect 0x7f5ff813c480 2024-03-25T18:24:13.307 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.306+0000 7f5fef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.312+0000 7f5fef7fe640 10 client.?.objecter ms_handle_connect 0x7f5ff80b4ff0 2024-03-25T18:24:13.313 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.312+0000 7f5fef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.319+0000 7f5fef7fe640 10 client.?.objecter ms_handle_connect 0x7f5ff80b4670 2024-03-25T18:24:13.320 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.319+0000 7f5fef7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:13.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.322+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676650478 0 0) 0x7f5ff8138100 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.322+0000 7f5fef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.322+0000 7f5fef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.322+0000 7f5fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.322+0000 7f5fef7fe640 10 monclient(hunting): my global_id is 4648 2024-03-25T18:24:13.324 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.322+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd8004eb0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.325+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 860142743 0 0) 0x7f5ff8132bf0 con 0x7f5ff813c480 2024-03-25T18:24:13.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.325+0000 7f5fef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.325+0000 7f5fef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.325+0000 7f5fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.325+0000 7f5fef7fe640 10 monclient(hunting): my global_id is 4644 2024-03-25T18:24:13.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.325+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ff8138100 con 0x7f5ff813c480 2024-03-25T18:24:13.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.332+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2674943223 0 0) 0x7f5fd8004eb0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.333 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.332+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5ff8132bf0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.332+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5fe0001950 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.332+0000 7f5fef7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.332+0000 7f5fef7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.332+0000 7f5fef7fe640 10 monclient(hunting): dump: 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:13.334 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:13.335 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:13.335 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.332+0000 7f5fef7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:13.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.337+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1013033843 0 0) 0x7f5ff8143620 con 0x7f5ff80b4670 2024-03-25T18:24:13.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.337+0000 7f5fef7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:13.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.337+0000 7f5fef7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:13.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.337+0000 7f5fef7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:13.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.337+0000 7f5fef7fe640 10 monclient(hunting): my global_id is 4676 2024-03-25T18:24:13.338 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.337+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fd8004eb0 con 0x7f5ff80b4670 2024-03-25T18:24:13.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1031212944 0 0) 0x7f5ff8132bf0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 >> v1:172.21.15.7:6790/0 conn(0x7f5ff80b4670 legacy=0x7f5ff8143b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 >> v1:172.21.15.7:6789/0 conn(0x7f5ff813c480 legacy=0x7f5ff81324e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:13.346 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 1 monclient: found mon.b 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ff8144250 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5fef7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 5 monclient: authenticate success, global_id 4648 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 10 monclient: _renew_subs 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ff8138520 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 10 client.4648.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 10 monclient: _renew_subs 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ff813e9b0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.347 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.345+0000 7f5ffd27d640 1 librados: init done 2024-03-25T18:24:13.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.346+0000 7f5ffd27d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:13.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.346+0000 7f5ffd27d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:13.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.346+0000 7f5ffd27d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:13.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.346+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ff81437b0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.628+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5fe0001d60 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.628+0000 7f5fef7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:13.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.628+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5fe0004980 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.628+0000 7f5fef7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:13.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.628+0000 7f5fef7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:13.629 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.628+0000 7f5fef7fe640 10 monclient: dump: 2024-03-25T18:24:13.629 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:13.630 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.628+0000 7f5ff5ffb640 4 set_mon_vals no callback set 2024-03-25T18:24:13.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.629+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f5fe0004ba0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.629+0000 7f5ff6ffd640 10 monclient: get_auth_request con 0x7f5fd806de00 auth_method 0 2024-03-25T18:24:13.631 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.630+0000 7f5fef7fe640 10 client.4648.objecter ms_handle_connect 0x7f5fd806de00 2024-03-25T18:24:13.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1056705622 0 0) 0x7f5fe0054800 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 10 client.4648.objecter ms_dispatch 0x7f5ff8000b90 osd_map(10..10 src has 1..10) v4 2024-03-25T18:24:13.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 3 client.4648.objecter handle_osd_map got epochs [10,10] > 0 2024-03-25T18:24:13.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 3 client.4648.objecter handle_osd_map decoding full epoch 10 2024-03-25T18:24:13.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 20 client.4648.objecter dump_active .. 0 homeless 2024-03-25T18:24:13.634 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5fe0050140 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:13.635 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.633+0000 7f5fef7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:13.906 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.905+0000 7f5ffd27d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:13.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.905+0000 7f5ffd27d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:13.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.905+0000 7f5ffd27d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:13.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.905+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ff81443e0 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.933+0000 7f5fef7fe640 1 -- 172.21.15.7:0/3498439326 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 2699920195) 0x7f5fe0052240 con 0x7f5ff80b4ff0 2024-03-25T18:24:13.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.933+0000 7f5fef7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:13.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.933+0000 7f5fef7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:13.934 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:13.934 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":10,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:14:16.162679+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:14:16.162679+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 10 librados: watch_flush enter 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 10 librados: watch_flush exit 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 20 client.4648.objecter shutdown clearing up homeless session... 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 10 client.4648.objecter successfully canceled tick 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 >> v1:172.21.15.7:6800/1717564767 conn(0x7f5fd806de00 legacy=0x7f5fd80702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 10 monclient: shutdown 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 >> v1:172.21.15.183:6789/0 conn(0x7f5ff80b4ff0 legacy=0x7f5ff8131dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 shutdown_connections 2024-03-25T18:24:13.937 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 >> 172.21.15.7:0/3498439326 conn(0x7f5ff80617a0 msgr2=0x7f5ff8061b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:13.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 shutdown_connections 2024-03-25T18:24:13.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 1 -- 172.21.15.7:0/3498439326 wait complete. 2024-03-25T18:24:13.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:13.935+0000 7f5ffd27d640 1 librados: shutdown 2024-03-25T18:24:13.948 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:14.636 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:21.272 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.644 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:21.281 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.684 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:21.320 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.698 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:21.324 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.704 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:21.341 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.705 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:21.331 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.717 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:21.354 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.728 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:21.354 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:14.740 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:21.377 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:20.755 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:20.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.870+0000 7f3da4225640 10 monclient: get_monmap_and_config 2024-03-25T18:24:20.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.870+0000 7f3da4225640 10 monclient: build_initial_monmap 2024-03-25T18:24:20.871 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.870+0000 7f3da4225640 10 monclient: monmap: 2024-03-25T18:24:20.871 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:20.871212+0000 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:20.871212+0000 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:20.872 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.871+0000 7f3da4225640 1 Processor -- start 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.871+0000 7f3da4225640 1 -- start start 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.871+0000 7f3da4225640 10 monclient: init 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 10 monclient(hunting): picked mon.noname-b con 0x7f3d9c0b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 10 monclient(hunting): picked mon.noname-c con 0x7f3d9c0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 10 monclient(hunting): picked mon.noname-a con 0x7f3d9c13c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0ab0e0 con 0x7f3d9c13c480 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0ab3c0 con 0x7f3d9c0b4670 2024-03-25T18:24:20.874 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0a9f30 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:20.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.872+0000 7f3da4225640 10 monclient(hunting): authenticate will time out at 7902.977114s 2024-03-25T18:24:20.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.885+0000 7f3da1fc1640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f3d9c0b4ff0 0x7f3d9c13bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:33496/0 (socket says 172.21.15.7:33496) 2024-03-25T18:24:20.886 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.885+0000 7f3da1fc1640 1 -- 172.21.15.7:0/2323868082 learned_addr learned my addr 172.21.15.7:0/2323868082 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:20.902 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.901+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 96076350 0 0) 0x7f3d9c0ab3c0 con 0x7f3d9c0b4670 2024-03-25T18:24:20.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.901+0000 7f3da0fbf640 10 monclient(hunting): _negotiate 2024-03-25T18:24:20.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.901+0000 7f3da0fbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:20.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.901+0000 7f3da0fbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:20.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.901+0000 7f3da0fbf640 10 monclient(hunting): my global_id is 4651 2024-03-25T18:24:20.903 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.901+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d88003520 con 0x7f3d9c0b4670 2024-03-25T18:24:20.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.907+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1586422928 0 0) 0x7f3d9c0a9f30 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.907+0000 7f3da0fbf640 10 monclient(hunting): _negotiate 2024-03-25T18:24:20.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.907+0000 7f3da0fbf640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:20.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.907+0000 7f3da0fbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:20.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.907+0000 7f3da0fbf640 10 monclient(hunting): my global_id is 4679 2024-03-25T18:24:20.909 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.907+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d9c0ab3c0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.910+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3157540829 0 0) 0x7f3d88003520 con 0x7f3d9c0b4670 2024-03-25T18:24:20.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.910+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 >> v1:172.21.15.7:6790/0 conn(0x7f3d9c0b4ff0 legacy=0x7f3d9c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:20.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.910+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 >> v1:172.21.15.7:6789/0 conn(0x7f3d9c13c480 legacy=0x7f3d9c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 10 monclient: _finish_hunting 0 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 1 monclient: found mon.noname-c 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0a9b70 con 0x7f3d9c0b4670 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 10 monclient: _finish_auth 0 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 10 monclient: _check_auth_tickets 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3d8c001690 con 0x7f3d9c0b4670 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:20.913 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da0fbf640 10 monclient: dump: 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da4225640 5 monclient: authenticate success, global_id 4651 2024-03-25T18:24:20.914 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.911+0000 7f3da4225640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:20.933 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d8c0017f0 con 0x7f3d9c0b4670 2024-03-25T18:24:20.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da0fbf640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:20.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da0fbf640 1 -- 172.21.15.7:0/2323868082 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3d8c001c00 con 0x7f3d9c0b4670 2024-03-25T18:24:20.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da0fbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:20.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da0fbf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:20.934 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da0fbf640 10 monclient: dump: 2024-03-25T18:24:20.934 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:20.934 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da4225640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:20.935 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.932+0000 7f3da4225640 4 set_mon_vals no callback set 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.933+0000 7f3da4225640 10 monclient: shutdown 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.933+0000 7f3da4225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.933+0000 7f3da4225640 1 -- 172.21.15.7:0/2323868082 >> v1:172.21.15.183:6789/0 conn(0x7f3d9c0b4670 legacy=0x7f3d9c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.933+0000 7f3da4225640 1 -- 172.21.15.7:0/2323868082 shutdown_connections 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.933+0000 7f3da4225640 1 -- 172.21.15.7:0/2323868082 >> 172.21.15.7:0/2323868082 conn(0x7f3d9c0617a0 msgr2=0x7f3d9c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.933+0000 7f3da4225640 1 -- 172.21.15.7:0/2323868082 shutdown_connections 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.934+0000 7f3da4225640 1 -- 172.21.15.7:0/2323868082 wait complete. 2024-03-25T18:24:20.938 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.934+0000 7f3da4225640 10 monclient: build_initial_monmap 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.934+0000 7f3da4225640 10 monclient: monmap: 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.935+0000 7f3da4225640 1 librados: starting msgr at 2024-03-25T18:24:20.939 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.935+0000 7f3da4225640 1 librados: starting objecter 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.935+0000 7f3da4225640 1 Processor -- start 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.935+0000 7f3da4225640 1 -- start start 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.935+0000 7f3da4225640 1 librados: setting wanted keys 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.935+0000 7f3da4225640 1 librados: calling monclient init 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.935+0000 7f3da4225640 10 monclient: init 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 10 monclient(hunting): picked mon.noname-c con 0x7f3d9c0b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 10 monclient(hunting): picked mon.noname-b con 0x7f3d9c0b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 10 monclient(hunting): picked mon.noname-a con 0x7f3d9c13c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0b05a0 con 0x7f3d9c13c480 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0b0880 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d9c0b02c0 con 0x7f3d9c0b4670 2024-03-25T18:24:20.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:20.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.936+0000 7f3da4225640 10 monclient(hunting): authenticate will time out at 7903.040870s 2024-03-25T18:24:20.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.946+0000 7f3da1fc1640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f3d9c0b4670 0x7f3d9c0b0ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:33562/0 (socket says 172.21.15.7:33562) 2024-03-25T18:24:20.947 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.946+0000 7f3da1fc1640 1 -- 172.21.15.7:0/2503853770 learned_addr learned my addr 172.21.15.7:0/2503853770 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:20.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.946+0000 7f3d927fc640 10 client.?.objecter ms_handle_connect 0x7f3d9c0b4670 2024-03-25T18:24:20.948 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.946+0000 7f3d927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:20.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.948+0000 7f3d927fc640 10 client.?.objecter ms_handle_connect 0x7f3d9c0b4ff0 2024-03-25T18:24:20.949 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.948+0000 7f3d927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:20.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.950+0000 7f3d927fc640 10 client.?.objecter ms_handle_connect 0x7f3d9c13c480 2024-03-25T18:24:20.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.950+0000 7f3d927fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:20.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.959+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717572652 0 0) 0x7f3d9c0b0880 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.960+0000 7f3d927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:20.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.960+0000 7f3d927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:20.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.960+0000 7f3d927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:20.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.960+0000 7f3d927fc640 10 monclient(hunting): my global_id is 4654 2024-03-25T18:24:20.961 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.960+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d80004eb0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.965 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.964+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1305932670 0 0) 0x7f3d9c0b02c0 con 0x7f3d9c0b4670 2024-03-25T18:24:20.965 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.964+0000 7f3d927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:20.965 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.964+0000 7f3d927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:20.965 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.964+0000 7f3d927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:20.965 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.964+0000 7f3d927fc640 10 monclient(hunting): my global_id is 4682 2024-03-25T18:24:20.965 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.964+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d9c0b0880 con 0x7f3d9c0b4670 2024-03-25T18:24:20.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.966+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3175149872 0 0) 0x7f3d9c0b05a0 con 0x7f3d9c13c480 2024-03-25T18:24:20.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.966+0000 7f3d927fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:20.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.966+0000 7f3d927fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:20.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.966+0000 7f3d927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:20.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.966+0000 7f3d927fc640 10 monclient(hunting): my global_id is 4650 2024-03-25T18:24:20.967 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.966+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d9c0b02c0 con 0x7f3d9c13c480 2024-03-25T18:24:20.977 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.976+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 83911941 0 0) 0x7f3d80004eb0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.976+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d9c0b05a0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.976+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3d8c001bc0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.976+0000 7f3d927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.976+0000 7f3d927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.976+0000 7f3d927fc640 10 monclient(hunting): dump: 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:20.978 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:20.979 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:20.979 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:20.979 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.976+0000 7f3d927fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.983+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2644807880 0 0) 0x7f3d9c0b02c0 con 0x7f3d9c13c480 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.983+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d80004eb0 con 0x7f3d9c13c480 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.983+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3d94001a60 con 0x7f3d9c13c480 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.983+0000 7f3d927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.983+0000 7f3d927fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.983+0000 7f3d927fc640 10 monclient(hunting): dump: 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.985 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.986 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:20.986 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:20.986 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:20.986 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:20.986 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:20.986 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.985+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3936465359 0 0) 0x7f3d9c0b0880 con 0x7f3d9c0b4670 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.985+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3d9c0b02c0 con 0x7f3d9c0b4670 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.985+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3d980019f0 con 0x7f3d9c0b4670 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.985+0000 7f3d927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.985+0000 7f3d927fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.985+0000 7f3d927fc640 10 monclient(hunting): dump: 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:20.987 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:20.988 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:20.988 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:20.988 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:20.988 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:20.988 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:20.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 636424130 0 0) 0x7f3d9c0b05a0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 >> v1:172.21.15.7:6790/0 conn(0x7f3d9c0b4670 legacy=0x7f3d9c0b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 >> v1:172.21.15.7:6789/0 conn(0x7f3d9c13c480 legacy=0x7f3d9c0aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 1 monclient: found mon.b 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d9c0b13b0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3d927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 5 monclient: authenticate success, global_id 4654 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 10 monclient: _renew_subs 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:20.990 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d9c0b1660 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 10 client.4654.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 10 monclient: _renew_subs 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d9c1e3780 con 0x7f3d9c0b4ff0 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.986+0000 7f3da4225640 1 librados: init done 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.987+0000 7f3da4225640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.987+0000 7f3da4225640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.987+0000 7f3da4225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:20.991 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:20.987+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d9c076320 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.270+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3d8c0038a0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.270+0000 7f3d927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.270+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f3d8c003bb0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.270+0000 7f3d927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.270+0000 7f3d927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.270+0000 7f3d927fc640 10 monclient: dump: 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:21.272 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:21.273 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.270+0000 7f3d93fff640 4 set_mon_vals no callback set 2024-03-25T18:24:21.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.271+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f3d8c019b80 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.272+0000 7f3da1fc1640 10 monclient: get_auth_request con 0x7f3d8006e0f0 auth_method 0 2024-03-25T18:24:21.274 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.272+0000 7f3d927fc640 10 client.4654.objecter ms_handle_connect 0x7f3d8006e0f0 2024-03-25T18:24:21.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.275+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7f3d8c0536e0 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.275+0000 7f3d927fc640 10 client.4654.objecter ms_dispatch 0x7f3d9c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:21.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.275+0000 7f3d927fc640 3 client.4654.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:21.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.275+0000 7f3d927fc640 3 client.4654.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:21.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.276+0000 7f3d927fc640 20 client.4654.objecter dump_active .. 0 homeless 2024-03-25T18:24:21.277 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.276+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3d8c04f020 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.276+0000 7f3d927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:21.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.276+0000 7f3d927fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:21.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.528+0000 7f3da4225640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:21.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.528+0000 7f3da4225640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:21.529 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.528+0000 7f3da4225640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:21.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.528+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d9c0ae320 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.555+0000 7f3d927fc640 1 -- 172.21.15.7:0/2503853770 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7f3d9c0ae320 con 0x7f3d9c0b4ff0 2024-03-25T18:24:21.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.555+0000 7f3d927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:21.556 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.555+0000 7f3d927fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:21.557 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:21.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 10 librados: watch_flush enter 2024-03-25T18:24:21.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 10 librados: watch_flush exit 2024-03-25T18:24:21.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 20 client.4654.objecter shutdown clearing up homeless session... 2024-03-25T18:24:21.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 10 client.4654.objecter successfully canceled tick 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 >> v1:172.21.15.7:6800/1717564767 conn(0x7f3d8006e0f0 legacy=0x7f3d800705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 10 monclient: shutdown 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 >> v1:172.21.15.183:6789/0 conn(0x7f3d9c0b4ff0 legacy=0x7f3d9c0ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 shutdown_connections 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 >> 172.21.15.7:0/2503853770 conn(0x7f3d9c0617a0 msgr2=0x7f3d9c0b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 shutdown_connections 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 1 -- 172.21.15.7:0/2503853770 wait complete. 2024-03-25T18:24:21.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:21.557+0000 7f3da4225640 1 librados: shutdown 2024-03-25T18:24:21.570 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:28.377 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.490+0000 7f04eed96640 10 monclient: get_monmap_and_config 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.490+0000 7f04eed96640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.490+0000 7f04eed96640 10 monclient: monmap: 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:28.491819+0000 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:28.491819+0000 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:28.492 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:28.493 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:28.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.491+0000 7f04eed96640 1 Processor -- start 2024-03-25T18:24:28.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.491+0000 7f04eed96640 1 -- start start 2024-03-25T18:24:28.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.491+0000 7f04eed96640 10 monclient: init 2024-03-25T18:24:28.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:28.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 10 monclient(hunting): picked mon.noname-c con 0x7f04e80b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:28.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 10 monclient(hunting): picked mon.noname-b con 0x7f04e80b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:28.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 10 monclient(hunting): picked mon.noname-a con 0x7f04e813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:28.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e80ab0e0 con 0x7f04e813c480 2024-03-25T18:24:28.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e80ab3c0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e80a9f30 con 0x7f04e80b4670 2024-03-25T18:24:28.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.495 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.492+0000 7f04eed96640 10 monclient(hunting): authenticate will time out at 7910.597469s 2024-03-25T18:24:28.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.495+0000 7f04ecb32640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f04e80b4ff0 0x7f04e813bd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:44110/0 (socket says 172.21.15.7:44110) 2024-03-25T18:24:28.496 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.495+0000 7f04ecb32640 1 -- 172.21.15.7:0/4245320282 learned_addr learned my addr 172.21.15.7:0/4245320282 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:28.506 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.505+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2576738887 0 0) 0x7f04e80ab3c0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.505+0000 7f04e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.505+0000 7f04e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.505+0000 7f04e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.505+0000 7f04e77fe640 10 monclient(hunting): my global_id is 4657 2024-03-25T18:24:28.507 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.505+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04d8003520 con 0x7f04e80b4ff0 2024-03-25T18:24:28.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.513+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720774213 0 0) 0x7f04e80a9f30 con 0x7f04e80b4670 2024-03-25T18:24:28.514 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.513+0000 7f04e77fe640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.513+0000 7f04e77fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.513+0000 7f04e77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.513+0000 7f04e77fe640 10 monclient(hunting): my global_id is 4685 2024-03-25T18:24:28.515 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.513+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04e80ab3c0 con 0x7f04e80b4670 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2889562552 0 0) 0x7f04d8003520 con 0x7f04e80b4ff0 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 >> v1:172.21.15.7:6790/0 conn(0x7f04e80b4670 legacy=0x7f04e81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 >> v1:172.21.15.7:6789/0 conn(0x7f04e813c480 legacy=0x7f04e814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 1 monclient: found mon.noname-c 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e80a9b70 con 0x7f04e80b4ff0 2024-03-25T18:24:28.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04dc001690 con 0x7f04e80b4ff0 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04e77fe640 10 monclient: dump: 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:28.520 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:28.521 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:28.521 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:28.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04eed96640 5 monclient: authenticate success, global_id 4657 2024-03-25T18:24:28.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.517+0000 7f04eed96640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:28.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04dc0017f0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04e77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04e77fe640 1 -- 172.21.15.7:0/4245320282 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04dc001c00 con 0x7f04e80b4ff0 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04eed96640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04eed96640 4 set_mon_vals no callback set 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04e77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04e77fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.525+0000 7f04e77fe640 10 monclient: dump: 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.527 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:28.528 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:28.528 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:28.528 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:28.528 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:28.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.526+0000 7f04eed96640 10 monclient: shutdown 2024-03-25T18:24:28.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.526+0000 7f04eed96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:28.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.526+0000 7f04eed96640 1 -- 172.21.15.7:0/4245320282 >> v1:172.21.15.183:6789/0 conn(0x7f04e80b4ff0 legacy=0x7f04e813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.530 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.526+0000 7f04eed96640 1 -- 172.21.15.7:0/4245320282 shutdown_connections 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.526+0000 7f04eed96640 1 -- 172.21.15.7:0/4245320282 >> 172.21.15.7:0/4245320282 conn(0x7f04e80617a0 msgr2=0x7f04e8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.526+0000 7f04eed96640 1 -- 172.21.15.7:0/4245320282 shutdown_connections 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.526+0000 7f04eed96640 1 -- 172.21.15.7:0/4245320282 wait complete. 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 10 monclient: build_initial_monmap 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 10 monclient: monmap: 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:28.531 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 1 librados: starting msgr at 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 1 librados: starting objecter 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 1 Processor -- start 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 1 -- start start 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 1 librados: setting wanted keys 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 1 librados: calling monclient init 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.527+0000 7f04eed96640 10 monclient: init 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 10 monclient(hunting): picked mon.noname-b con 0x7f04e80b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 10 monclient(hunting): picked mon.noname-a con 0x7f04e80b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:28.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 10 monclient(hunting): picked mon.noname-c con 0x7f04e813c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e8137a70 con 0x7f04e80b4ff0 2024-03-25T18:24:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e8137d50 con 0x7f04e80b4670 2024-03-25T18:24:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04e8143410 con 0x7f04e813c480 2024-03-25T18:24:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:28.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.528+0000 7f04eed96640 10 monclient(hunting): authenticate will time out at 7910.633252s 2024-03-25T18:24:28.537 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.536+0000 7f04ed333640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f04e813c480 0x7f04e81454a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:38110/0 (socket says 172.21.15.7:38110) 2024-03-25T18:24:28.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.536+0000 7f04ed333640 1 -- 172.21.15.7:0/1334768797 learned_addr learned my addr 172.21.15.7:0/1334768797 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:28.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.537+0000 7f04e57fa640 10 client.?.objecter ms_handle_connect 0x7f04e80b4670 2024-03-25T18:24:28.538 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.537+0000 7f04e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.537+0000 7f04e57fa640 10 client.?.objecter ms_handle_connect 0x7f04e813c480 2024-03-25T18:24:28.539 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.537+0000 7f04e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.542+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2758984496 0 0) 0x7f04e8137d50 con 0x7f04e80b4670 2024-03-25T18:24:28.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.542+0000 7f04e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.543 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.542+0000 7f04e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.544 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.542+0000 7f04e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.544 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.542+0000 7f04e57fa640 10 monclient(hunting): my global_id is 4660 2024-03-25T18:24:28.544 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.542+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc004eb0 con 0x7f04e80b4670 2024-03-25T18:24:28.550 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.549+0000 7f04e57fa640 10 client.?.objecter ms_handle_connect 0x7f04e80b4ff0 2024-03-25T18:24:28.550 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.549+0000 7f04e57fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:28.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.550+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367974211 0 0) 0x7f04cc004eb0 con 0x7f04e80b4670 2024-03-25T18:24:28.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.550+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e8137d50 con 0x7f04e80b4670 2024-03-25T18:24:28.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.550+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04dc001bc0 con 0x7f04e80b4670 2024-03-25T18:24:28.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.550+0000 7f04e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.550+0000 7f04e57fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:28.551 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.550+0000 7f04e57fa640 10 monclient(hunting): dump: 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:28.552 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.550+0000 7f04e57fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:28.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.564+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 484572020 0 0) 0x7f04e8137a70 con 0x7f04e80b4ff0 2024-03-25T18:24:28.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.564+0000 7f04e57fa640 10 monclient(hunting): _negotiate 2024-03-25T18:24:28.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.564+0000 7f04e57fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:28.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.564+0000 7f04e57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:28.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.564+0000 7f04e57fa640 10 monclient(hunting): my global_id is 4653 2024-03-25T18:24:28.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.564+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f04cc004eb0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.576+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2183422689 0 0) 0x7f04cc004eb0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.576+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f04e8137a70 con 0x7f04e80b4ff0 2024-03-25T18:24:28.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.576+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04d4001930 con 0x7f04e80b4ff0 2024-03-25T18:24:28.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.576+0000 7f04e57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.577 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.576+0000 7f04e57fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.576+0000 7f04e57fa640 10 monclient(hunting): dump: 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:28.578 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:28.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.590+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3256885228 0 0) 0x7f04e8137a70 con 0x7f04e80b4ff0 2024-03-25T18:24:28.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.590+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 >> v1:172.21.15.7:6790/0 conn(0x7f04e813c480 legacy=0x7f04e81454a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.590+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 >> v1:172.21.15.183:6789/0 conn(0x7f04e80b4670 legacy=0x7f04e8143c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:28.592 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.590+0000 7f04e57fa640 10 monclient: _finish_hunting 0 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.590+0000 7f04e57fa640 1 monclient: found mon.a 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04e57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04e57fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f04e81436f0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04e57fa640 10 monclient: _finish_auth 0 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04e57fa640 10 monclient: _check_auth_tickets 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04e57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 5 monclient: authenticate success, global_id 4653 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 10 monclient: _renew_subs 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f04e8145bb0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 10 client.4653.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 10 monclient: _renew_subs 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:24:28.593 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f04e8145fd0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.591+0000 7f04eed96640 1 librados: init done 2024-03-25T18:24:28.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.592+0000 7f04eed96640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.592+0000 7f04eed96640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.592+0000 7f04eed96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:24:28.594 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.592+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f04e8143930 con 0x7f04e80b4ff0 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.908+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f04d40035f0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.908+0000 7f04e57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.908+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f04d4001d80 con 0x7f04e80b4ff0 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.908+0000 7f04e57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.908+0000 7f04e57fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.908+0000 7f04e57fa640 10 monclient: dump: 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:28.910 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.908+0000 7f04e6ffd640 4 set_mon_vals no callback set 2024-03-25T18:24:28.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.909+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f04d4019690 con 0x7f04e80b4ff0 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.910+0000 7f04ecb32640 10 monclient: get_auth_request con 0x7f04cc06de10 auth_method 0 2024-03-25T18:24:28.912 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.910+0000 7f04e57fa640 10 client.4653.objecter ms_handle_connect 0x7f04cc06de10 2024-03-25T18:24:28.916 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7f04d40533e0 con 0x7f04e80b4ff0 2024-03-25T18:24:28.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 10 client.4653.objecter ms_dispatch 0x7f04e8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:28.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 3 client.4653.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:28.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 3 client.4653.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:28.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 20 client.4653.objecter dump_active .. 0 homeless 2024-03-25T18:24:28.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f04d404ed20 con 0x7f04e80b4ff0 2024-03-25T18:24:28.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:28.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:28.915+0000 7f04e57fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:29.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.207+0000 7f04eed96640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.207+0000 7f04eed96640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.207+0000 7f04eed96640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:24:29.208 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.207+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f04e806dbd0 con 0x7f04e80b4ff0 2024-03-25T18:24:29.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.254+0000 7f04e57fa640 1 -- 172.21.15.7:0/1334768797 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7f04d4050e20 con 0x7f04e80b4ff0 2024-03-25T18:24:29.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.254+0000 7f04e57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:29.256 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.254+0000 7f04e57fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:29.256 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:29.256 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:29.258 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.256+0000 7f04eed96640 10 librados: watch_flush enter 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 10 librados: watch_flush exit 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 20 client.4653.objecter shutdown clearing up homeless session... 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 10 client.4653.objecter successfully canceled tick 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 >> v1:172.21.15.7:6800/1717564767 conn(0x7f04cc06de10 legacy=0x7f04cc0702d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 10 monclient: shutdown 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 >> v1:172.21.15.7:6789/0 conn(0x7f04e80b4ff0 legacy=0x7f04e8144d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 shutdown_connections 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 >> 172.21.15.7:0/1334768797 conn(0x7f04e80617a0 msgr2=0x7f04e8140670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 shutdown_connections 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 1 -- 172.21.15.7:0/1334768797 wait complete. 2024-03-25T18:24:29.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:29.257+0000 7f04eed96640 1 librados: shutdown 2024-03-25T18:24:29.270 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:34.441 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:41.078 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.492 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:41.129 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.522 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:10:41.159 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.558 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:41.184 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.566 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:41.192 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.576 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:41.213 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.596 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:41.233 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.599 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:10:41.226 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:34.625 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:10:41.262 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:36.076 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:36.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.189+0000 7f1836086640 10 monclient: get_monmap_and_config 2024-03-25T18:24:36.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.189+0000 7f1836086640 10 monclient: build_initial_monmap 2024-03-25T18:24:36.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.189+0000 7f1836086640 10 monclient: monmap: 2024-03-25T18:24:36.190 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:36.190511+0000 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:36.190511+0000 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:36.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.190+0000 7f1836086640 1 Processor -- start 2024-03-25T18:24:36.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.190+0000 7f1836086640 1 -- start start 2024-03-25T18:24:36.192 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.190+0000 7f1836086640 10 monclient: init 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 10 monclient(hunting): picked mon.noname-c con 0x7f18300b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 10 monclient(hunting): picked mon.noname-a con 0x7f18300b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 10 monclient(hunting): picked mon.noname-b con 0x7f183013c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300ab0e0 con 0x7f18300b4670 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300ab3c0 con 0x7f18300b4ff0 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18300a9f30 con 0x7f183013c480 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:36.193 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.191+0000 7f1836086640 10 monclient(hunting): authenticate will time out at 7918.296281s 2024-03-25T18:24:36.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.197+0000 7f182ffff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f183013c480 0x7f183014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:34416/0 (socket says 172.21.15.7:34416) 2024-03-25T18:24:36.198 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.197+0000 7f182ffff640 1 -- 172.21.15.7:0/3311379835 learned_addr learned my addr 172.21.15.7:0/3311379835 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:36.210 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.209+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2121891054 0 0) 0x7f18300ab3c0 con 0x7f18300b4ff0 2024-03-25T18:24:36.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.209+0000 7f182e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.209+0000 7f182e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.209+0000 7f182e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.209+0000 7f182e7fc640 10 monclient(hunting): my global_id is 4663 2024-03-25T18:24:36.211 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.209+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1818003520 con 0x7f18300b4ff0 2024-03-25T18:24:36.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.212+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 880341997 0 0) 0x7f18300a9f30 con 0x7f183013c480 2024-03-25T18:24:36.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.212+0000 7f182e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.212+0000 7f182e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.212+0000 7f182e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.212+0000 7f182e7fc640 10 monclient(hunting): my global_id is 4688 2024-03-25T18:24:36.213 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.212+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18300ab3c0 con 0x7f183013c480 2024-03-25T18:24:36.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.224+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2309914343 0 0) 0x7f18300ab0e0 con 0x7f18300b4670 2024-03-25T18:24:36.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.224+0000 7f182e7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.224+0000 7f182e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.224+0000 7f182e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.224+0000 7f182e7fc640 10 monclient(hunting): my global_id is 4656 2024-03-25T18:24:36.225 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.224+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18300a9f30 con 0x7f18300b4670 2024-03-25T18:24:36.230 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1547388184 0 0) 0x7f18300ab3c0 con 0x7f183013c480 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 >> v1:172.21.15.183:6789/0 conn(0x7f18300b4ff0 legacy=0x7f183013bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 >> v1:172.21.15.7:6789/0 conn(0x7f18300b4670 legacy=0x7f18301425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 1 monclient: found mon.noname-b 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18300a9b70 con 0x7f183013c480 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f18280019f0 con 0x7f183013c480 2024-03-25T18:24:36.231 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f1836086640 5 monclient: authenticate success, global_id 4688 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f1836086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f182e7fc640 10 monclient: dump: 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:36.232 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:36.233 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:36.233 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.229+0000 7f1836086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:36.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1828001b50 con 0x7f183013c480 2024-03-25T18:24:36.242 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f182e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f182e7fc640 1 -- 172.21.15.7:0/3311379835 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1828001f60 con 0x7f183013c480 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f182e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f182e7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f182e7fc640 10 monclient: dump: 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:36.243 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:36.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f1836086640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:36.244 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.241+0000 7f1836086640 4 set_mon_vals no callback set 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.242+0000 7f1836086640 10 monclient: shutdown 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.242+0000 7f1836086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.242+0000 7f1836086640 1 -- 172.21.15.7:0/3311379835 >> v1:172.21.15.7:6790/0 conn(0x7f183013c480 legacy=0x7f183014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.242+0000 7f1836086640 1 -- 172.21.15.7:0/3311379835 shutdown_connections 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.242+0000 7f1836086640 1 -- 172.21.15.7:0/3311379835 >> 172.21.15.7:0/3311379835 conn(0x7f18300617a0 msgr2=0x7f1830061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.242+0000 7f1836086640 1 -- 172.21.15.7:0/3311379835 shutdown_connections 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.242+0000 7f1836086640 1 -- 172.21.15.7:0/3311379835 wait complete. 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 10 monclient: build_initial_monmap 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 10 monclient: monmap: 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:36.246 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 1 librados: starting msgr at 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 1 librados: starting objecter 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 1 Processor -- start 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 1 -- start start 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 1 librados: setting wanted keys 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 1 librados: calling monclient init 2024-03-25T18:24:36.247 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.243+0000 7f1836086640 10 monclient: init 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 10 monclient(hunting): picked mon.noname-c con 0x7f18300b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 10 monclient(hunting): picked mon.noname-a con 0x7f18300b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 10 monclient(hunting): picked mon.noname-b con 0x7f183013c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1830143d20 con 0x7f18300b4ff0 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1830144a80 con 0x7f183013c480 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1830144d60 con 0x7f18300b4670 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:36.248 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.244+0000 7f1836086640 10 monclient(hunting): authenticate will time out at 7918.349173s 2024-03-25T18:24:36.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.250+0000 7f182ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f183013c480 0x7f18301e2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:40542/0 (socket says 172.21.15.7:40542) 2024-03-25T18:24:36.251 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.250+0000 7f182ffff640 1 -- 172.21.15.7:0/3391430183 learned_addr learned my addr 172.21.15.7:0/3391430183 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:36.253 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.252+0000 7f1817fff640 10 client.?.objecter ms_handle_connect 0x7f183013c480 2024-03-25T18:24:36.253 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.252+0000 7f1817fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.255+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2371107540 0 0) 0x7f1830144a80 con 0x7f183013c480 2024-03-25T18:24:36.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.255+0000 7f1817fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.255+0000 7f1817fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.255+0000 7f1817fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.255+0000 7f1817fff640 10 monclient(hunting): my global_id is 4666 2024-03-25T18:24:36.257 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.255+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c004eb0 con 0x7f183013c480 2024-03-25T18:24:36.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.258+0000 7f1817fff640 10 client.?.objecter ms_handle_connect 0x7f18300b4ff0 2024-03-25T18:24:36.259 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.258+0000 7f1817fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.261+0000 7f1817fff640 10 client.?.objecter ms_handle_connect 0x7f18300b4670 2024-03-25T18:24:36.262 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.261+0000 7f1817fff640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:36.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.265+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3468653886 0 0) 0x7f181c004eb0 con 0x7f183013c480 2024-03-25T18:24:36.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.265+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1830144a80 con 0x7f183013c480 2024-03-25T18:24:36.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.265+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1828001f20 con 0x7f183013c480 2024-03-25T18:24:36.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.265+0000 7f1817fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.265+0000 7f1817fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:36.266 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.265+0000 7f1817fff640 10 monclient(hunting): dump: 2024-03-25T18:24:36.266 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:36.267 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.265+0000 7f1817fff640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:36.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.267+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2390452690 0 0) 0x7f1830144d60 con 0x7f18300b4670 2024-03-25T18:24:36.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.267+0000 7f1817fff640 10 monclient(hunting): _negotiate 2024-03-25T18:24:36.268 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.267+0000 7f1817fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:36.269 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.267+0000 7f1817fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:36.269 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.267+0000 7f1817fff640 10 monclient(hunting): my global_id is 4691 2024-03-25T18:24:36.269 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.267+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f181c004eb0 con 0x7f18300b4670 2024-03-25T18:24:36.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3477995280 0 0) 0x7f1830144a80 con 0x7f183013c480 2024-03-25T18:24:36.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 >> v1:172.21.15.7:6790/0 conn(0x7f18300b4670 legacy=0x7f18301457d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 >> v1:172.21.15.7:6789/0 conn(0x7f18300b4ff0 legacy=0x7f1830145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 10 monclient: _finish_hunting 0 2024-03-25T18:24:36.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 1 monclient: found mon.b 2024-03-25T18:24:36.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:36.279 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1830145040 con 0x7f183013c480 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 10 monclient: _finish_auth 0 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 10 monclient: _check_auth_tickets 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1817fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 5 monclient: authenticate success, global_id 4666 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 10 monclient: _renew_subs 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1830143f90 con 0x7f183013c480 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 10 client.4666.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 10 monclient: _renew_subs 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f183013dcb0 con 0x7f183013c480 2024-03-25T18:24:36.280 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.278+0000 7f1836086640 1 librados: init done 2024-03-25T18:24:36.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.279+0000 7f1836086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.279+0000 7f1836086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.279+0000 7f1836086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:36.281 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.279+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f183006dbd0 con 0x7f183013c480 2024-03-25T18:24:36.559 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.558+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f18280014f0 con 0x7f183013c480 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.558+0000 7f1817fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.558+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1828004fb0 con 0x7f183013c480 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.558+0000 7f1817fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.558+0000 7f1817fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.558+0000 7f1817fff640 10 monclient: dump: 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:36.560 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:36.561 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:36.561 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:36.561 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:36.561 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:36.561 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.558+0000 7f182dffb640 4 set_mon_vals no callback set 2024-03-25T18:24:36.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.559+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f182801af80 con 0x7f183013c480 2024-03-25T18:24:36.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.560+0000 7f182f7fe640 10 monclient: get_auth_request con 0x7f181c06dd90 auth_method 0 2024-03-25T18:24:36.562 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.560+0000 7f1817fff640 10 client.4666.objecter ms_handle_connect 0x7f181c06dd90 2024-03-25T18:24:36.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.563+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7f1828054b10 con 0x7f183013c480 2024-03-25T18:24:36.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.563+0000 7f1817fff640 10 client.4666.objecter ms_dispatch 0x7f1830000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:36.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.563+0000 7f1817fff640 3 client.4666.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:36.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.563+0000 7f1817fff640 3 client.4666.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:36.565 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.563+0000 7f1817fff640 20 client.4666.objecter dump_active .. 0 homeless 2024-03-25T18:24:36.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.564+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1828050450 con 0x7f183013c480 2024-03-25T18:24:36.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.564+0000 7f1817fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:36.566 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.564+0000 7f1817fff640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.806+0000 7f1836086640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.806+0000 7f1836086640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.806+0000 7f1836086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:36.807 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.806+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f18300b3290 con 0x7f183013c480 2024-03-25T18:24:36.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.839+0000 7f1817fff640 1 -- 172.21.15.7:0/3391430183 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7f1828052550 con 0x7f183013c480 2024-03-25T18:24:36.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.839+0000 7f1817fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:36.840 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.839+0000 7f1817fff640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:36.841 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:36.841 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 10 librados: watch_flush enter 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 10 librados: watch_flush exit 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 20 client.4666.objecter shutdown clearing up homeless session... 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 10 client.4666.objecter successfully canceled tick 2024-03-25T18:24:36.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 >> v1:172.21.15.7:6800/1717564767 conn(0x7f181c06dd90 legacy=0x7f181c070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 10 monclient: shutdown 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.841+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 >> v1:172.21.15.183:6789/0 conn(0x7f183013c480 legacy=0x7f18301e2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.842+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 shutdown_connections 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.842+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 >> 172.21.15.7:0/3391430183 conn(0x7f18300617a0 msgr2=0x7f1830141ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.842+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 shutdown_connections 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.842+0000 7f1836086640 1 -- 172.21.15.7:0/3391430183 wait complete. 2024-03-25T18:24:36.844 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:36.842+0000 7f1836086640 1 librados: shutdown 2024-03-25T18:24:36.854 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:43.660 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:43.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.768+0000 7fbb6bef6640 10 monclient: get_monmap_and_config 2024-03-25T18:24:43.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.768+0000 7fbb6bef6640 10 monclient: build_initial_monmap 2024-03-25T18:24:43.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.768+0000 7fbb6bef6640 10 monclient: monmap: 2024-03-25T18:24:43.769 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:43.769 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:43.769 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:43.769060+0000 2024-03-25T18:24:43.770 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:43.769060+0000 2024-03-25T18:24:43.770 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:43.770 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:43.770 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:43.770 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:43.770 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:43.770 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:43.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.768+0000 7fbb6bef6640 1 Processor -- start 2024-03-25T18:24:43.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.768+0000 7fbb6bef6640 1 -- start start 2024-03-25T18:24:43.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.768+0000 7fbb6bef6640 10 monclient: init 2024-03-25T18:24:43.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:43.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:24:43.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 10 monclient(hunting): picked mon.noname-b con 0x7fbb640b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:43.771 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 10 monclient(hunting): picked mon.noname-a con 0x7fbb640b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:43.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 10 monclient(hunting): picked mon.noname-c con 0x7fbb6413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:43.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640ab0e0 con 0x7fbb640b4670 2024-03-25T18:24:43.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640ab3c0 con 0x7fbb6413c480 2024-03-25T18:24:43.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640a9f30 con 0x7fbb640b4ff0 2024-03-25T18:24:43.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:43.772 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.769+0000 7fbb6bef6640 10 monclient(hunting): authenticate will time out at 7925.874474s 2024-03-25T18:24:43.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.783+0000 7fbb69491640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7fbb640b4670 0x7fbb641425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:45132/0 (socket says 172.21.15.7:45132) 2024-03-25T18:24:43.785 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.784+0000 7fbb69491640 1 -- 172.21.15.7:0/513994411 learned_addr learned my addr 172.21.15.7:0/513994411 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:43.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3314828308 0 0) 0x7fbb640a9f30 con 0x7fbb640b4ff0 2024-03-25T18:24:43.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): _negotiate 2024-03-25T18:24:43.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:43.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:43.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): my global_id is 4694 2024-03-25T18:24:43.803 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb50003520 con 0x7fbb640b4ff0 2024-03-25T18:24:43.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2149705824 0 0) 0x7fbb640ab3c0 con 0x7fbb6413c480 2024-03-25T18:24:43.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): _negotiate 2024-03-25T18:24:43.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:43.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:43.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 10 monclient(hunting): my global_id is 4669 2024-03-25T18:24:43.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.802+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb640a9f30 con 0x7fbb6413c480 2024-03-25T18:24:43.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.810+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1027386373 0 0) 0x7fbb640ab0e0 con 0x7fbb640b4670 2024-03-25T18:24:43.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.810+0000 7fbb68c90640 10 monclient(hunting): _negotiate 2024-03-25T18:24:43.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.810+0000 7fbb68c90640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:43.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.810+0000 7fbb68c90640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:43.811 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.810+0000 7fbb68c90640 10 monclient(hunting): my global_id is 4662 2024-03-25T18:24:43.812 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.810+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb640ab3c0 con 0x7fbb640b4670 2024-03-25T18:24:43.819 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 340601605 0 0) 0x7fbb640a9f30 con 0x7fbb6413c480 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 >> v1:172.21.15.7:6790/0 conn(0x7fbb640b4ff0 legacy=0x7fbb6413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 >> v1:172.21.15.7:6789/0 conn(0x7fbb640b4670 legacy=0x7fbb641425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 10 monclient: _finish_hunting 0 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 1 monclient: found mon.noname-c 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640a9b70 con 0x7fbb6413c480 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 10 monclient: _finish_auth 0 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 10 monclient: _check_auth_tickets 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbb5c0019f0 con 0x7fbb6413c480 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb68c90640 10 monclient: dump: 2024-03-25T18:24:43.820 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb6bef6640 5 monclient: authenticate success, global_id 4669 2024-03-25T18:24:43.821 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.818+0000 7fbb6bef6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:43.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb5c001b50 con 0x7fbb6413c480 2024-03-25T18:24:43.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb68c90640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:43.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb68c90640 1 -- 172.21.15.7:0/513994411 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbb5c001fa0 con 0x7fbb6413c480 2024-03-25T18:24:43.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb6bef6640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb6bef6640 4 set_mon_vals no callback set 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb68c90640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb68c90640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.852+0000 7fbb68c90640 10 monclient: dump: 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:43.854 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:43.855 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:43.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.853+0000 7fbb6bef6640 10 monclient: shutdown 2024-03-25T18:24:43.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.853+0000 7fbb6bef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:43.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.853+0000 7fbb6bef6640 1 -- 172.21.15.7:0/513994411 >> v1:172.21.15.183:6789/0 conn(0x7fbb6413c480 legacy=0x7fbb6414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:43.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.853+0000 7fbb6bef6640 1 -- 172.21.15.7:0/513994411 shutdown_connections 2024-03-25T18:24:43.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.853+0000 7fbb6bef6640 1 -- 172.21.15.7:0/513994411 >> 172.21.15.7:0/513994411 conn(0x7fbb640617a0 msgr2=0x7fbb64061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:43.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.853+0000 7fbb6bef6640 1 -- 172.21.15.7:0/513994411 shutdown_connections 2024-03-25T18:24:43.856 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.853+0000 7fbb6bef6640 1 -- 172.21.15.7:0/513994411 wait complete. 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.854+0000 7fbb6bef6640 10 monclient: build_initial_monmap 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.854+0000 7fbb6bef6640 10 monclient: monmap: 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.855+0000 7fbb6bef6640 1 librados: starting msgr at 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.855+0000 7fbb6bef6640 1 librados: starting objecter 2024-03-25T18:24:43.857 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.855+0000 7fbb6bef6640 1 Processor -- start 2024-03-25T18:24:43.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.855+0000 7fbb6bef6640 1 -- start start 2024-03-25T18:24:43.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.855+0000 7fbb6bef6640 1 librados: setting wanted keys 2024-03-25T18:24:43.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.855+0000 7fbb6bef6640 1 librados: calling monclient init 2024-03-25T18:24:43.858 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.855+0000 7fbb6bef6640 10 monclient: init 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 10 monclient(hunting): picked mon.noname-a con 0x7fbb640b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 10 monclient(hunting): picked mon.noname-c con 0x7fbb640b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 10 monclient(hunting): picked mon.noname-b con 0x7fbb6413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640ae230 con 0x7fbb640b4670 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640b02d0 con 0x7fbb6413c480 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb640b05b0 con 0x7fbb640b4ff0 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:43.859 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.856+0000 7fbb6bef6640 10 monclient(hunting): authenticate will time out at 7925.961207s 2024-03-25T18:24:43.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.869+0000 7fbb6a493640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fbb6413c480 0x7fbb640aedd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:43218/0 (socket says 172.21.15.7:43218) 2024-03-25T18:24:43.870 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.869+0000 7fbb6a493640 1 -- 172.21.15.7:0/1615068176 learned_addr learned my addr 172.21.15.7:0/1615068176 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:43.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.874+0000 7fbb5a7fc640 10 client.?.objecter ms_handle_connect 0x7fbb640b4670 2024-03-25T18:24:43.875 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.874+0000 7fbb5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:43.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.878+0000 7fbb5a7fc640 10 client.?.objecter ms_handle_connect 0x7fbb6413c480 2024-03-25T18:24:43.879 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.878+0000 7fbb5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:43.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.882+0000 7fbb5a7fc640 10 client.?.objecter ms_handle_connect 0x7fbb640b4ff0 2024-03-25T18:24:43.883 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.882+0000 7fbb5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:43.889 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.888+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 826294645 0 0) 0x7fbb640b02d0 con 0x7fbb6413c480 2024-03-25T18:24:43.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.888+0000 7fbb5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:43.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.888+0000 7fbb5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:43.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.888+0000 7fbb5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:43.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.888+0000 7fbb5a7fc640 10 monclient(hunting): my global_id is 4672 2024-03-25T18:24:43.890 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.888+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb48004eb0 con 0x7fbb6413c480 2024-03-25T18:24:43.897 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.896+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2850127823 0 0) 0x7fbb640b05b0 con 0x7fbb640b4ff0 2024-03-25T18:24:43.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.896+0000 7fbb5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:43.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.896+0000 7fbb5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:43.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.896+0000 7fbb5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:43.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.896+0000 7fbb5a7fc640 10 monclient(hunting): my global_id is 4697 2024-03-25T18:24:43.898 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.896+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb640b02d0 con 0x7fbb640b4ff0 2024-03-25T18:24:43.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.906+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253670325 0 0) 0x7fbb640ae230 con 0x7fbb640b4670 2024-03-25T18:24:43.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.906+0000 7fbb5a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:24:43.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.906+0000 7fbb5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:43.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.906+0000 7fbb5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:43.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.906+0000 7fbb5a7fc640 10 monclient(hunting): my global_id is 4665 2024-03-25T18:24:43.907 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.906+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb640b05b0 con 0x7fbb640b4670 2024-03-25T18:24:43.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.909+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3946444511 0 0) 0x7fbb48004eb0 con 0x7fbb6413c480 2024-03-25T18:24:43.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.909+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbb640ae230 con 0x7fbb6413c480 2024-03-25T18:24:43.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.909+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbb5c001fa0 con 0x7fbb6413c480 2024-03-25T18:24:43.910 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.909+0000 7fbb5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.909+0000 7fbb5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.909+0000 7fbb5a7fc640 10 monclient(hunting): dump: 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:43.911 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.909+0000 7fbb5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:43.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1272091296 0 0) 0x7fbb640ae230 con 0x7fbb6413c480 2024-03-25T18:24:43.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 >> v1:172.21.15.7:6790/0 conn(0x7fbb640b4ff0 legacy=0x7fbb640ae6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:43.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 >> v1:172.21.15.7:6789/0 conn(0x7fbb640b4670 legacy=0x7fbb640b1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:43.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:24:43.917 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 1 monclient: found mon.b 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb640b0890 con 0x7fbb6413c480 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb6bef6640 5 monclient: authenticate success, global_id 4672 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb6bef6640 10 monclient: _renew_subs 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb6bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.915+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb640b1730 con 0x7fbb6413c480 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.916+0000 7fbb6bef6640 10 client.4672.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.916+0000 7fbb6bef6640 10 monclient: _renew_subs 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.916+0000 7fbb6bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.916+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb641e3780 con 0x7fbb6413c480 2024-03-25T18:24:43.918 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.916+0000 7fbb6bef6640 1 librados: init done 2024-03-25T18:24:43.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.917+0000 7fbb6bef6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:43.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.917+0000 7fbb6bef6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:43.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.917+0000 7fbb6bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:43.919 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:43.917+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb640688c0 con 0x7fbb6413c480 2024-03-25T18:24:44.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.183+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbb5c003c20 con 0x7fbb6413c480 2024-03-25T18:24:44.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.183+0000 7fbb5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:44.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.183+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fbb5c003ff0 con 0x7fbb6413c480 2024-03-25T18:24:44.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.183+0000 7fbb5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:44.184 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.183+0000 7fbb5a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.183+0000 7fbb5a7fc640 10 monclient: dump: 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:44.185 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.183+0000 7fbb5bfff640 4 set_mon_vals no callback set 2024-03-25T18:24:44.186 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.184+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fbb5c004210 con 0x7fbb6413c480 2024-03-25T18:24:44.186 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.184+0000 7fbb69c92640 10 monclient: get_auth_request con 0x7fbb4806de20 auth_method 0 2024-03-25T18:24:44.186 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.185+0000 7fbb5a7fc640 10 client.4672.objecter ms_handle_connect 0x7fbb4806de20 2024-03-25T18:24:44.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.189+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7fbb5c054c20 con 0x7fbb6413c480 2024-03-25T18:24:44.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.189+0000 7fbb5a7fc640 10 client.4672.objecter ms_dispatch 0x7fbb64000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:44.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.189+0000 7fbb5a7fc640 3 client.4672.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:44.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.189+0000 7fbb5a7fc640 3 client.4672.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:44.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.189+0000 7fbb5a7fc640 20 client.4672.objecter dump_active .. 0 homeless 2024-03-25T18:24:44.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.190+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbb5c050730 con 0x7fbb6413c480 2024-03-25T18:24:44.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.190+0000 7fbb5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:44.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.190+0000 7fbb5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:44.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.449+0000 7fbb6bef6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:44.450 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.449+0000 7fbb6bef6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:44.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.449+0000 7fbb6bef6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:44.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.449+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbb640ae3e0 con 0x7fbb6413c480 2024-03-25T18:24:44.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.481+0000 7fbb5a7fc640 1 -- 172.21.15.7:0/1615068176 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7fbb5c052660 con 0x7fbb6413c480 2024-03-25T18:24:44.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.481+0000 7fbb5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:44.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.481+0000 7fbb5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:44.483 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:44.483 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.483+0000 7fbb6bef6640 10 librados: watch_flush enter 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.483+0000 7fbb6bef6640 10 librados: watch_flush exit 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.483+0000 7fbb6bef6640 20 client.4672.objecter shutdown clearing up homeless session... 2024-03-25T18:24:44.485 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.483+0000 7fbb6bef6640 10 client.4672.objecter successfully canceled tick 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.483+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 >> v1:172.21.15.7:6800/1717564767 conn(0x7fbb4806de20 legacy=0x7fbb480702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 10 monclient: shutdown 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 >> v1:172.21.15.183:6789/0 conn(0x7fbb6413c480 legacy=0x7fbb640aedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 shutdown_connections 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 >> 172.21.15.7:0/1615068176 conn(0x7fbb640617a0 msgr2=0x7fbb64145420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 shutdown_connections 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 1 -- 172.21.15.7:0/1615068176 wait complete. 2024-03-25T18:24:44.486 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:44.484+0000 7fbb6bef6640 1 librados: shutdown 2024-03-25T18:24:44.497 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:51.303 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.420+0000 7f48fa40e640 10 monclient: get_monmap_and_config 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.420+0000 7f48fa40e640 10 monclient: build_initial_monmap 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.420+0000 7f48fa40e640 10 monclient: monmap: 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:51.421840+0000 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:51.421840+0000 2024-03-25T18:24:51.422 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:51.423 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:51.423 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:51.423 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:51.423 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:51.423 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:51.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.421+0000 7f48fa40e640 1 Processor -- start 2024-03-25T18:24:51.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.421+0000 7f48fa40e640 1 -- start start 2024-03-25T18:24:51.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.421+0000 7f48fa40e640 10 monclient: init 2024-03-25T18:24:51.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:51.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:24:51.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 10 monclient(hunting): picked mon.noname-c con 0x7f48f40b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:51.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 10 monclient(hunting): picked mon.noname-a con 0x7f48f40b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:51.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 10 monclient(hunting): picked mon.noname-b con 0x7f48f413c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:51.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f40ab0e0 con 0x7f48f40b4670 2024-03-25T18:24:51.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f40ab3c0 con 0x7f48f40b4ff0 2024-03-25T18:24:51.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f40a9f30 con 0x7f48f413c480 2024-03-25T18:24:51.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:51.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.422+0000 7f48fa40e640 10 monclient(hunting): authenticate will time out at 7933.527154s 2024-03-25T18:24:51.438 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.437+0000 7f48f89ab640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f48f413c480 0x7f48f414dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:56094/0 (socket says 172.21.15.7:56094) 2024-03-25T18:24:51.438 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.437+0000 7f48f89ab640 1 -- 172.21.15.7:0/2253685659 learned_addr learned my addr 172.21.15.7:0/2253685659 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:51.451 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.450+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1101309186 0 0) 0x7f48f40a9f30 con 0x7f48f413c480 2024-03-25T18:24:51.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.450+0000 7f48f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.450+0000 7f48f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.450+0000 7f48f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.450+0000 7f48f2ffd640 10 monclient(hunting): my global_id is 4700 2024-03-25T18:24:51.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.450+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48dc003520 con 0x7f48f413c480 2024-03-25T18:24:51.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.452+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3620300517 0 0) 0x7f48f40ab0e0 con 0x7f48f40b4670 2024-03-25T18:24:51.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.452+0000 7f48f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.452+0000 7f48f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.452+0000 7f48f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.452+0000 7f48f2ffd640 10 monclient(hunting): my global_id is 4668 2024-03-25T18:24:51.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.452+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f40a9f30 con 0x7f48f40b4670 2024-03-25T18:24:51.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.458+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3760025231 0 0) 0x7f48f40ab3c0 con 0x7f48f40b4ff0 2024-03-25T18:24:51.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.458+0000 7f48f2ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.458+0000 7f48f2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.458+0000 7f48f2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.458+0000 7f48f2ffd640 10 monclient(hunting): my global_id is 4675 2024-03-25T18:24:51.459 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.458+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f40ab0e0 con 0x7f48f40b4ff0 2024-03-25T18:24:51.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 124937237 0 0) 0x7f48dc003520 con 0x7f48f413c480 2024-03-25T18:24:51.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 >> v1:172.21.15.183:6789/0 conn(0x7f48f40b4ff0 legacy=0x7f48f413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 >> v1:172.21.15.7:6789/0 conn(0x7f48f40b4670 legacy=0x7f48f41425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:51.467 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 1 monclient: found mon.noname-b 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f40a9b70 con 0x7f48f413c480 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 10 monclient: _finish_auth 0 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f48ec001a10 con 0x7f48f413c480 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48f2ffd640 10 monclient: dump: 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.468 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:51.469 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:51.469 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:51.469 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:51.469 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:51.469 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:51.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48fa40e640 5 monclient: authenticate success, global_id 4700 2024-03-25T18:24:51.469 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.466+0000 7f48fa40e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:51.473 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.472+0000 7f48f2ffd640 1 -- 172.21.15.7:0/2253685659 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48ec001b70 con 0x7f48f413c480 2024-03-25T18:24:51.473 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.472+0000 7f48f2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:51.473 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.472+0000 7f48fa40e640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:51.473 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.472+0000 7f48fa40e640 4 set_mon_vals no callback set 2024-03-25T18:24:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.473+0000 7f48fa40e640 10 monclient: shutdown 2024-03-25T18:24:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.473+0000 7f48fa40e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.473+0000 7f48fa40e640 1 -- 172.21.15.7:0/2253685659 >> v1:172.21.15.7:6790/0 conn(0x7f48f413c480 legacy=0x7f48f414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.473+0000 7f48fa40e640 1 -- 172.21.15.7:0/2253685659 shutdown_connections 2024-03-25T18:24:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.473+0000 7f48fa40e640 1 -- 172.21.15.7:0/2253685659 >> 172.21.15.7:0/2253685659 conn(0x7f48f40617a0 msgr2=0x7f48f4061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.473+0000 7f48fa40e640 1 -- 172.21.15.7:0/2253685659 shutdown_connections 2024-03-25T18:24:51.475 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.473+0000 7f48fa40e640 1 -- 172.21.15.7:0/2253685659 wait complete. 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.474+0000 7f48fa40e640 10 monclient: build_initial_monmap 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.474+0000 7f48fa40e640 10 monclient: monmap: 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:51.476 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:51.477 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:51.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.475+0000 7f48fa40e640 1 librados: starting msgr at 2024-03-25T18:24:51.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.475+0000 7f48fa40e640 1 librados: starting objecter 2024-03-25T18:24:51.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.475+0000 7f48fa40e640 1 Processor -- start 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.475+0000 7f48fa40e640 1 -- start start 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.475+0000 7f48fa40e640 1 librados: setting wanted keys 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.475+0000 7f48fa40e640 1 librados: calling monclient init 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.475+0000 7f48fa40e640 10 monclient: init 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 10 monclient(hunting): picked mon.noname-a con 0x7f48f40b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 10 monclient(hunting): picked mon.noname-b con 0x7f48f40b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 10 monclient(hunting): picked mon.noname-c con 0x7f48f413c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f41439d0 con 0x7f48f40b4670 2024-03-25T18:24:51.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f4144490 con 0x7f48f40b4ff0 2024-03-25T18:24:51.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f48f4144770 con 0x7f48f413c480 2024-03-25T18:24:51.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:51.479 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.476+0000 7f48fa40e640 10 monclient(hunting): authenticate will time out at 7933.581144s 2024-03-25T18:24:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.491+0000 7f48f37fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f48f40b4ff0 0x7f48f4145580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:59610/0 (socket says 172.21.15.7:59610) 2024-03-25T18:24:51.492 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.491+0000 7f48f37fe640 1 -- 172.21.15.7:0/2024115730 learned_addr learned my addr 172.21.15.7:0/2024115730 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.493+0000 7f48f0ff9640 10 client.?.objecter ms_handle_connect 0x7f48f40b4ff0 2024-03-25T18:24:51.494 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.493+0000 7f48f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:51.498 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.497+0000 7f48f0ff9640 10 client.?.objecter ms_handle_connect 0x7f48f413c480 2024-03-25T18:24:51.499 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.497+0000 7f48f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:51.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.500+0000 7f48f0ff9640 10 client.?.objecter ms_handle_connect 0x7f48f40b4670 2024-03-25T18:24:51.501 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.500+0000 7f48f0ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:51.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.502+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1338848200 0 0) 0x7f48f4144490 con 0x7f48f40b4ff0 2024-03-25T18:24:51.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.502+0000 7f48f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.502+0000 7f48f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.502+0000 7f48f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.502+0000 7f48f0ff9640 10 monclient(hunting): my global_id is 4678 2024-03-25T18:24:51.503 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.502+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48e0004eb0 con 0x7f48f40b4ff0 2024-03-25T18:24:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.504+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1574457863 0 0) 0x7f48f4144770 con 0x7f48f413c480 2024-03-25T18:24:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.504+0000 7f48f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.504+0000 7f48f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.504+0000 7f48f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.504+0000 7f48f0ff9640 10 monclient(hunting): my global_id is 4703 2024-03-25T18:24:51.505 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.504+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f4144490 con 0x7f48f413c480 2024-03-25T18:24:51.508 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.507+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4053884190 0 0) 0x7f48f41439d0 con 0x7f48f40b4670 2024-03-25T18:24:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.507+0000 7f48f0ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.507+0000 7f48f0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.507+0000 7f48f0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.507+0000 7f48f0ff9640 10 monclient(hunting): my global_id is 4671 2024-03-25T18:24:51.509 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.507+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f48f4144770 con 0x7f48f40b4670 2024-03-25T18:24:51.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.517+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1092515545 0 0) 0x7f48f4144490 con 0x7f48f413c480 2024-03-25T18:24:51.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.517+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48f41439d0 con 0x7f48f413c480 2024-03-25T18:24:51.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.517+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f48ec001e80 con 0x7f48f413c480 2024-03-25T18:24:51.518 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.517+0000 7f48f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.517+0000 7f48f0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.517+0000 7f48f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:51.519 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.517+0000 7f48f0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.519+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2778677444 0 0) 0x7f48e0004eb0 con 0x7f48f40b4ff0 2024-03-25T18:24:51.520 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.519+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48f4144490 con 0x7f48f40b4ff0 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.519+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f48e8001a30 con 0x7f48f40b4ff0 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.519+0000 7f48f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.519+0000 7f48f0ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.519+0000 7f48f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:51.521 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:51.526 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.525+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1732080978 0 0) 0x7f48f4144770 con 0x7f48f40b4670 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.525+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f48e0004eb0 con 0x7f48f40b4670 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.525+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f48e4001830 con 0x7f48f40b4670 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.525+0000 7f48f0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.525+0000 7f48f0ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.525+0000 7f48f0ff9640 10 monclient(hunting): dump: 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:51.527 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 726355608 0 0) 0x7f48f41439d0 con 0x7f48f413c480 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 >> v1:172.21.15.183:6789/0 conn(0x7f48f40b4ff0 legacy=0x7f48f4145580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 >> v1:172.21.15.7:6789/0 conn(0x7f48f40b4670 legacy=0x7f48f4144e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 1 monclient: found mon.c 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48f4144a50 con 0x7f48f413c480 2024-03-25T18:24:51.532 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48f0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 5 monclient: authenticate success, global_id 4703 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 10 monclient: _renew_subs 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48f41466e0 con 0x7f48f413c480 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 10 client.4703.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 10 monclient: _renew_subs 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f48f4132920 con 0x7f48f413c480 2024-03-25T18:24:51.533 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.530+0000 7f48fa40e640 1 librados: init done 2024-03-25T18:24:51.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.531+0000 7f48fa40e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:51.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.531+0000 7f48fa40e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:51.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.531+0000 7f48fa40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:24:51.534 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.531+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48f4143820 con 0x7f48f413c480 2024-03-25T18:24:51.860 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.859+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f48ec001510 con 0x7f48f413c480 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.859+0000 7f48f0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.859+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f48ec004e10 con 0x7f48f413c480 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.859+0000 7f48f0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.859+0000 7f48f0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.859+0000 7f48f0ff9640 10 monclient: dump: 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:51.861 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:51.862 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:51.862 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:51.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.859+0000 7f48f27fc640 4 set_mon_vals no callback set 2024-03-25T18:24:51.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.860+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f48ec01ae20 con 0x7f48f413c480 2024-03-25T18:24:51.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.860+0000 7f48f3fff640 10 monclient: get_auth_request con 0x7f48e006e0f0 auth_method 0 2024-03-25T18:24:51.862 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.861+0000 7f48f0ff9640 10 client.4703.objecter ms_handle_connect 0x7f48e006e0f0 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.865+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7f48ec054a90 con 0x7f48f413c480 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.865+0000 7f48f0ff9640 10 client.4703.objecter ms_dispatch 0x7f48f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.865+0000 7f48f0ff9640 3 client.4703.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.865+0000 7f48f0ff9640 3 client.4703.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.865+0000 7f48f0ff9640 20 client.4703.objecter dump_active .. 0 homeless 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.866+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f48ec0503d0 con 0x7f48f413c480 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.866+0000 7f48f0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:51.867 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:51.866+0000 7f48f0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.133+0000 7f48fa40e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.133+0000 7f48fa40e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.133+0000 7f48fa40e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.7:6790/0 2024-03-25T18:24:52.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.133+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 --> v1:172.21.15.7:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f48f406dbd0 con 0x7f48f413c480 2024-03-25T18:24:52.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.159+0000 7f48f0ff9640 1 -- 172.21.15.7:0/2024115730 <== mon.2 v1:172.21.15.7:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7f48ec0524d0 con 0x7f48f413c480 2024-03-25T18:24:52.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.159+0000 7f48f0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:52.160 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.159+0000 7f48f0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:52.160 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:52.160 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 10 librados: watch_flush enter 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 10 librados: watch_flush exit 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 20 client.4703.objecter shutdown clearing up homeless session... 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 10 client.4703.objecter successfully canceled tick 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 >> v1:172.21.15.7:6800/1717564767 conn(0x7f48e006e0f0 legacy=0x7f48e00705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 10 monclient: shutdown 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 >> v1:172.21.15.7:6790/0 conn(0x7f48f413c480 legacy=0x7f48f41e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 shutdown_connections 2024-03-25T18:24:52.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 >> 172.21.15.7:0/2024115730 conn(0x7f48f40617a0 msgr2=0x7f48f4140700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:52.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 shutdown_connections 2024-03-25T18:24:52.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 1 -- 172.21.15.7:0/2024115730 wait complete. 2024-03-25T18:24:52.164 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:52.161+0000 7f48fa40e640 1 librados: shutdown 2024-03-25T18:24:52.175 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:24:58.983 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:24:59.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.103+0000 7f6566539640 10 monclient: get_monmap_and_config 2024-03-25T18:24:59.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.103+0000 7f6566539640 10 monclient: build_initial_monmap 2024-03-25T18:24:59.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.103+0000 7f6566539640 10 monclient: monmap: 2024-03-25T18:24:59.104 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:59.104 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:24:59.104371+0000 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:24:59.104371+0000 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:24:59.105 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:59.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 1 Processor -- start 2024-03-25T18:24:59.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 1 -- start start 2024-03-25T18:24:59.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient: init 2024-03-25T18:24:59.106 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient(hunting): picked mon.noname-a con 0x7f65600b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient(hunting): picked mon.noname-c con 0x7f65600b44b0 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient(hunting): picked mon.noname-b con 0x7f65600b1b00 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600a96f0 con 0x7f65600b4e70 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65600ab0e0 con 0x7f65600b44b0 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6560145f70 con 0x7f65600b1b00 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:59.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.104+0000 7f6566539640 10 monclient(hunting): authenticate will time out at 7941.209552s 2024-03-25T18:24:59.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.118+0000 7f6564ad6640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f65600b1b00 0x7f656014dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:60654/0 (socket says 172.21.15.7:60654) 2024-03-25T18:24:59.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.118+0000 7f6564ad6640 1 -- 172.21.15.7:0/3963730761 learned_addr learned my addr 172.21.15.7:0/3963730761 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:59.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.132+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4209010115 0 0) 0x7f65600a96f0 con 0x7f65600b4e70 2024-03-25T18:24:59.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.132+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.132+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.132+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.132+0000 7f655effd640 10 monclient(hunting): my global_id is 4674 2024-03-25T18:24:59.134 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.132+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6548003520 con 0x7f65600b4e70 2024-03-25T18:24:59.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.133+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1425662863 0 0) 0x7f6560145f70 con 0x7f65600b1b00 2024-03-25T18:24:59.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.133+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.133+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.133+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.133+0000 7f655effd640 10 monclient(hunting): my global_id is 4706 2024-03-25T18:24:59.135 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.133+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f65600a96f0 con 0x7f65600b1b00 2024-03-25T18:24:59.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.138+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617764789 0 0) 0x7f65600ab0e0 con 0x7f65600b44b0 2024-03-25T18:24:59.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.138+0000 7f655effd640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.138+0000 7f655effd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.138+0000 7f655effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.138+0000 7f655effd640 10 monclient(hunting): my global_id is 4681 2024-03-25T18:24:59.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.138+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6560145f70 con 0x7f65600b44b0 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1706057493 0 0) 0x7f6548003520 con 0x7f65600b4e70 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 >> v1:172.21.15.7:6790/0 conn(0x7f65600b1b00 legacy=0x7f656014dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 >> v1:172.21.15.183:6789/0 conn(0x7f65600b44b0 legacy=0x7f6560145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 10 monclient: _finish_hunting 0 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 1 monclient: found mon.noname-a 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:24:59.145 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65600a9b80 con 0x7f65600b4e70 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 10 monclient: _finish_auth 0 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 10 monclient: _check_auth_tickets 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6550001690 con 0x7f65600b4e70 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f655effd640 10 monclient: dump: 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:59.146 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:59.147 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:59.147 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:59.147 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:59.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f6566539640 5 monclient: authenticate success, global_id 4674 2024-03-25T18:24:59.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.143+0000 7f6566539640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:24:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f65500017f0 con 0x7f65600b4e70 2024-03-25T18:24:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f655effd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f655effd640 1 -- 172.21.15.7:0/3963730761 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6550001c00 con 0x7f65600b4e70 2024-03-25T18:24:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f655effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f655effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:24:59.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f655effd640 10 monclient: dump: 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f6566539640 10 monclient: get_monmap_and_config success 2024-03-25T18:24:59.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.149+0000 7f6566539640 4 set_mon_vals no callback set 2024-03-25T18:24:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.150+0000 7f6566539640 10 monclient: shutdown 2024-03-25T18:24:59.152 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.150+0000 7f6566539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.150+0000 7f6566539640 1 -- 172.21.15.7:0/3963730761 >> v1:172.21.15.7:6789/0 conn(0x7f65600b4e70 legacy=0x7f656013f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.150+0000 7f6566539640 1 -- 172.21.15.7:0/3963730761 shutdown_connections 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.150+0000 7f6566539640 1 -- 172.21.15.7:0/3963730761 >> 172.21.15.7:0/3963730761 conn(0x7f65600617a0 msgr2=0x7f6560061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.150+0000 7f6566539640 1 -- 172.21.15.7:0/3963730761 shutdown_connections 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.150+0000 7f6566539640 1 -- 172.21.15.7:0/3963730761 wait complete. 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 10 monclient: build_initial_monmap 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 10 monclient: monmap: 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:24:59.153 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 1 librados: starting msgr at 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 1 librados: starting objecter 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 1 Processor -- start 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 1 -- start start 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 1 librados: setting wanted keys 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 1 librados: calling monclient init 2024-03-25T18:24:59.154 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.151+0000 7f6566539640 10 monclient: init 2024-03-25T18:24:59.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 10 monclient: _reopen_session rank -1 2024-03-25T18:24:59.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:24:59.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 10 monclient(hunting): picked mon.noname-b con 0x7f65600b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:24:59.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 10 monclient(hunting): picked mon.noname-c con 0x7f65600b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:24:59.155 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 10 monclient(hunting): picked mon.noname-a con 0x7f65600b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:24:59.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656013afa0 con 0x7f65600b4e70 2024-03-25T18:24:59.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f656012a310 con 0x7f65600b1b00 2024-03-25T18:24:59.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6560128810 con 0x7f65600b44b0 2024-03-25T18:24:59.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 10 monclient(hunting): _renew_subs 2024-03-25T18:24:59.156 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.152+0000 7f6566539640 10 monclient(hunting): authenticate will time out at 7941.257266s 2024-03-25T18:24:59.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.158+0000 7f655ffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f65600b1b00 0x7f656014e6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:35812/0 (socket says 172.21.15.7:35812) 2024-03-25T18:24:59.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.158+0000 7f655ffff640 1 -- 172.21.15.7:0/3061644703 learned_addr learned my addr 172.21.15.7:0/3061644703 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:24:59.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.162+0000 7f655cff9640 10 client.?.objecter ms_handle_connect 0x7f65600b1b00 2024-03-25T18:24:59.163 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.162+0000 7f655cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.165+0000 7f655cff9640 10 client.?.objecter ms_handle_connect 0x7f65600b44b0 2024-03-25T18:24:59.166 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.165+0000 7f655cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.168+0000 7f655cff9640 10 client.?.objecter ms_handle_connect 0x7f65600b4e70 2024-03-25T18:24:59.169 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.168+0000 7f655cff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:24:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.170+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342677864 0 0) 0x7f656012a310 con 0x7f65600b1b00 2024-03-25T18:24:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.170+0000 7f655cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.170+0000 7f655cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.170+0000 7f655cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.170+0000 7f655cff9640 10 monclient(hunting): my global_id is 4684 2024-03-25T18:24:59.171 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.170+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f654c004eb0 con 0x7f65600b1b00 2024-03-25T18:24:59.178 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.177+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2269012414 0 0) 0x7f656013afa0 con 0x7f65600b4e70 2024-03-25T18:24:59.178 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.177+0000 7f655cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.178 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.177+0000 7f655cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.178 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.177+0000 7f655cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.177+0000 7f655cff9640 10 monclient(hunting): my global_id is 4677 2024-03-25T18:24:59.179 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.177+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656012a310 con 0x7f65600b4e70 2024-03-25T18:24:59.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.180+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4050568618 0 0) 0x7f6560128810 con 0x7f65600b44b0 2024-03-25T18:24:59.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.180+0000 7f655cff9640 10 monclient(hunting): _negotiate 2024-03-25T18:24:59.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.180+0000 7f655cff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:24:59.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.180+0000 7f655cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:24:59.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.180+0000 7f655cff9640 10 monclient(hunting): my global_id is 4709 2024-03-25T18:24:59.182 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.180+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f656013afa0 con 0x7f65600b44b0 2024-03-25T18:24:59.189 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.188+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 363335387 0 0) 0x7f654c004eb0 con 0x7f65600b1b00 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.188+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6560128810 con 0x7f65600b1b00 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.188+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6550001bc0 con 0x7f65600b1b00 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.188+0000 7f655cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.188+0000 7f655cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.188+0000 7f655cff9640 10 monclient(hunting): dump: 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:59.190 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:59.191 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:59.191 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.188+0000 7f655cff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:24:59.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4260621496 0 0) 0x7f6560128810 con 0x7f65600b1b00 2024-03-25T18:24:59.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 >> v1:172.21.15.7:6790/0 conn(0x7f65600b44b0 legacy=0x7f65601d79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 >> v1:172.21.15.7:6789/0 conn(0x7f65600b4e70 legacy=0x7f65601e2e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.195 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 10 monclient: _finish_hunting 0 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 1 monclient: found mon.b 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656012a560 con 0x7f65600b1b00 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 10 monclient: _finish_auth 0 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 10 monclient: _check_auth_tickets 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f655cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 5 monclient: authenticate success, global_id 4684 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 10 monclient: _renew_subs 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f656012a840 con 0x7f65600b1b00 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 10 client.4684.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 10 monclient: _renew_subs 2024-03-25T18:24:59.196 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:59.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f656014efc0 con 0x7f65600b1b00 2024-03-25T18:24:59.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.194+0000 7f6566539640 1 librados: init done 2024-03-25T18:24:59.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.195+0000 7f6566539640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.195+0000 7f6566539640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.195+0000 7f6566539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:59.197 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.195+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f656004f730 con 0x7f65600b1b00 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.474+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6550001210 con 0x7f65600b1b00 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.474+0000 7f655cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.474+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f6550003a40 con 0x7f65600b1b00 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.474+0000 7f655cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.474+0000 7f655cff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.474+0000 7f655cff9640 10 monclient: dump: 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:24:59.476 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:24:59.477 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:24:59.477 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:24:59.477 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:24:59.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.474+0000 7f655e7fc640 4 set_mon_vals no callback set 2024-03-25T18:24:59.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.475+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f6550003c60 con 0x7f65600b1b00 2024-03-25T18:24:59.477 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.476+0000 7f655f7fe640 10 monclient: get_auth_request con 0x7f654c06dda0 auth_method 0 2024-03-25T18:24:59.478 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.476+0000 7f655cff9640 10 client.4684.objecter ms_handle_connect 0x7f654c06dda0 2024-03-25T18:24:59.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7f65500548a0 con 0x7f65600b1b00 2024-03-25T18:24:59.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 10 client.4684.objecter ms_dispatch 0x7f6560000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:24:59.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 3 client.4684.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:24:59.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 3 client.4684.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:24:59.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 20 client.4684.objecter dump_active .. 0 homeless 2024-03-25T18:24:59.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f65500501e0 con 0x7f65600b1b00 2024-03-25T18:24:59.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:24:59.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.480+0000 7f655cff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:24:59.533 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:06.170 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.596 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:06.233 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.627 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:06.264 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.653 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:06.279 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.657 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:06.283 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.659 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:06.296 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.664 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:06.301 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.682 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:06.308 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.692+0000 7f6566539640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:59.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.692+0000 7f6566539640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:59.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.692+0000 7f6566539640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:24:59.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.692+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f65600aa090 con 0x7f65600b1b00 2024-03-25T18:24:59.696 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:06.333 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:24:59.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.719+0000 7f655cff9640 1 -- 172.21.15.7:0/3061644703 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7f65500522e0 con 0x7f65600b1b00 2024-03-25T18:24:59.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.719+0000 7f655cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:24:59.720 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.719+0000 7f655cff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:24:59.721 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:24:59.721 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:24:59.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.721+0000 7f6566539640 10 librados: watch_flush enter 2024-03-25T18:24:59.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.721+0000 7f6566539640 10 librados: watch_flush exit 2024-03-25T18:24:59.723 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.721+0000 7f6566539640 20 client.4684.objecter shutdown clearing up homeless session... 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 10 client.4684.objecter successfully canceled tick 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 >> v1:172.21.15.7:6800/1717564767 conn(0x7f654c06dda0 legacy=0x7f654c070260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 10 monclient: shutdown 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 >> v1:172.21.15.183:6789/0 conn(0x7f65600b1b00 legacy=0x7f656014e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 shutdown_connections 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 >> 172.21.15.7:0/3061644703 conn(0x7f65600617a0 msgr2=0x7f65600b64f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 shutdown_connections 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 1 -- 172.21.15.7:0/3061644703 wait complete. 2024-03-25T18:24:59.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:24:59.722+0000 7f6566539640 1 librados: shutdown 2024-03-25T18:24:59.734 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:06.540 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:06.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.643+0000 7f1332b3a640 10 monclient: get_monmap_and_config 2024-03-25T18:25:06.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.643+0000 7f1332b3a640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.644 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.643+0000 7f1332b3a640 10 monclient: monmap: 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:25:06.644693+0000 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:25:06.644693+0000 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:25:06.645 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:06.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.644+0000 7f1332b3a640 1 Processor -- start 2024-03-25T18:25:06.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.644+0000 7f1332b3a640 1 -- start start 2024-03-25T18:25:06.646 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.644+0000 7f1332b3a640 10 monclient: init 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.644+0000 7f1332b3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.644+0000 7f1332b3a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 10 monclient(hunting): picked mon.noname-a con 0x7f132c0b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 10 monclient(hunting): picked mon.noname-b con 0x7f132c0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 10 monclient(hunting): picked mon.noname-c con 0x7f132c0b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c0a96f0 con 0x7f132c0b4e70 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c0ab0e0 con 0x7f132c0b1b00 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c145f70 con 0x7f132c0b44b0 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f1332b3a640 10 monclient(hunting): authenticate will time out at 7948.749726s 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f13310d7640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7f132c0b1b00 0x7f132c14dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:60650/0 (socket says 172.21.15.7:60650) 2024-03-25T18:25:06.647 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.645+0000 7f13310d7640 1 -- 172.21.15.7:0/288316666 learned_addr learned my addr 172.21.15.7:0/288316666 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:06.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.690+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2197017309 0 0) 0x7f132c0a96f0 con 0x7f132c0b4e70 2024-03-25T18:25:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.690+0000 7f132b7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.690+0000 7f132b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.690+0000 7f132b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.690+0000 7f132b7fe640 10 monclient(hunting): my global_id is 4680 2024-03-25T18:25:06.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.690+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f131c003520 con 0x7f132c0b4e70 2024-03-25T18:25:06.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 77405728 0 0) 0x7f131c003520 con 0x7f132c0b4e70 2024-03-25T18:25:06.701 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 >> v1:172.21.15.7:6790/0 conn(0x7f132c0b44b0 legacy=0x7f132c145860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 >> v1:172.21.15.183:6789/0 conn(0x7f132c0b1b00 legacy=0x7f132c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 1 monclient: found mon.noname-a 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.7:6789/0 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c0a9b80 con 0x7f132c0b4e70 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1320001690 con 0x7f132c0b4e70 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f132b7fe640 10 monclient: dump: 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:06.702 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f1332b3a640 5 monclient: authenticate success, global_id 4680 2024-03-25T18:25:06.703 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.700+0000 7f1332b3a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 <== mon.0 v1:172.21.15.7:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f13200017f0 con 0x7f132c0b4e70 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f132b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f132b7fe640 1 -- 172.21.15.7:0/288316666 <== mon.0 v1:172.21.15.7:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1320001c40 con 0x7f132c0b4e70 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f132b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f132b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f132b7fe640 10 monclient: dump: 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:06.713 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:06.714 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:06.714 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:06.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:06.714 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 4 set_mon_vals no callback set 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 10 monclient: shutdown 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 1 -- 172.21.15.7:0/288316666 >> v1:172.21.15.7:6789/0 conn(0x7f132c0b4e70 legacy=0x7f132c13f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 1 -- 172.21.15.7:0/288316666 shutdown_connections 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 1 -- 172.21.15.7:0/288316666 >> 172.21.15.7:0/288316666 conn(0x7f132c0617a0 msgr2=0x7f132c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 1 -- 172.21.15.7:0/288316666 shutdown_connections 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.712+0000 7f1332b3a640 1 -- 172.21.15.7:0/288316666 wait complete. 2024-03-25T18:25:06.715 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.713+0000 7f1332b3a640 10 monclient: build_initial_monmap 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.713+0000 7f1332b3a640 10 monclient: monmap: 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 librados: starting msgr at 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 librados: starting objecter 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 Processor -- start 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 -- start start 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 librados: setting wanted keys 2024-03-25T18:25:06.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 librados: calling monclient init 2024-03-25T18:25:06.717 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient: init 2024-03-25T18:25:06.717 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient(hunting): picked mon.noname-b con 0x7f132c0b1b00 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient(hunting): picked mon.noname-c con 0x7f132c0b44b0 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient(hunting): picked mon.noname-a con 0x7f132c0b4e70 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c1451c0 con 0x7f132c0b4e70 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c13b6e0 con 0x7f132c0b1b00 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f132c13cb50 con 0x7f132c0b44b0 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:06.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.714+0000 7f1332b3a640 10 monclient(hunting): authenticate will time out at 7948.819565s 2024-03-25T18:25:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.730+0000 7f13310d7640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f132c0b4e70 0x7f132c142790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:46310/0 (socket says 172.21.15.7:46310) 2024-03-25T18:25:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.730+0000 7f13310d7640 1 -- 172.21.15.7:0/2185903271 learned_addr learned my addr 172.21.15.7:0/2185903271 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.731+0000 7f13297fa640 10 client.?.objecter ms_handle_connect 0x7f132c0b1b00 2024-03-25T18:25:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.731+0000 7f13297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.733+0000 7f13297fa640 10 client.?.objecter ms_handle_connect 0x7f132c0b4e70 2024-03-25T18:25:06.734 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.733+0000 7f13297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.742 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.741+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2136398969 0 0) 0x7f132c1451c0 con 0x7f132c0b4e70 2024-03-25T18:25:06.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.741+0000 7f13297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.741+0000 7f13297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.741+0000 7f13297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.741+0000 7f13297fa640 10 monclient(hunting): my global_id is 4683 2024-03-25T18:25:06.743 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.741+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1310004eb0 con 0x7f132c0b4e70 2024-03-25T18:25:06.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.745+0000 7f13297fa640 10 client.?.objecter ms_handle_connect 0x7f132c0b44b0 2024-03-25T18:25:06.746 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.745+0000 7f13297fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:06.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.755+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2489843628 0 0) 0x7f132c13b6e0 con 0x7f132c0b1b00 2024-03-25T18:25:06.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.755+0000 7f13297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.755+0000 7f13297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.755+0000 7f13297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.755+0000 7f13297fa640 10 monclient(hunting): my global_id is 4690 2024-03-25T18:25:06.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.755+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f132c1451c0 con 0x7f132c0b1b00 2024-03-25T18:25:06.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.757+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034735681 0 0) 0x7f1310004eb0 con 0x7f132c0b4e70 2024-03-25T18:25:06.758 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.757+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f132c13b6e0 con 0x7f132c0b4e70 2024-03-25T18:25:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.763+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1959458536 0 0) 0x7f132c13cb50 con 0x7f132c0b44b0 2024-03-25T18:25:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.763+0000 7f13297fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.763+0000 7f13297fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.763+0000 7f13297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:06.764 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.763+0000 7f13297fa640 10 monclient(hunting): my global_id is 4715 2024-03-25T18:25:06.765 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.763+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1310004eb0 con 0x7f132c0b44b0 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.765+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973758792 0 0) 0x7f132c1451c0 con 0x7f132c0b1b00 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.765+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f132c13cb50 con 0x7f132c0b1b00 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.765+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1320001b80 con 0x7f132c0b1b00 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.765+0000 7f13297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.765+0000 7f13297fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.765+0000 7f13297fa640 10 monclient(hunting): dump: 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.767 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.768 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:06.768 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:06.768 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:06.768 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:06.768 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:06.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.765+0000 7f13297fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:06.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1324001c50 con 0x7f132c0b4e70 2024-03-25T18:25:06.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 10 monclient(hunting): dump: 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2620334160 0 0) 0x7f132c13b6e0 con 0x7f132c0b4e70 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 >> v1:172.21.15.7:6790/0 conn(0x7f132c0b44b0 legacy=0x7f132c143b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 >> v1:172.21.15.183:6789/0 conn(0x7f132c0b1b00 legacy=0x7f132c13aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 1 monclient: found mon.a 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c139f50 con 0x7f132c0b4e70 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f13297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 5 monclient: authenticate success, global_id 4683 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 10 monclient: _renew_subs 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f132c13bb70 con 0x7f132c0b4e70 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 10 client.4683.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 10 monclient: _renew_subs 2024-03-25T18:25:06.778 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:06.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f132c144210 con 0x7f132c0b4e70 2024-03-25T18:25:06.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.775+0000 7f1332b3a640 1 librados: init done 2024-03-25T18:25:06.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.776+0000 7f1332b3a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.776+0000 7f1332b3a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:06.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.776+0000 7f1332b3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:06.779 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:06.776+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f132c144460 con 0x7f132c0b4e70 2024-03-25T18:25:07.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.095+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1324004b60 con 0x7f132c0b4e70 2024-03-25T18:25:07.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.095+0000 7f13297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:07.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.095+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f1324004e10 con 0x7f132c0b4e70 2024-03-25T18:25:07.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.095+0000 7f13297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:07.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.095+0000 7f13297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:07.096 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.095+0000 7f13297fa640 10 monclient: dump: 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:07.097 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.095+0000 7f132affd640 4 set_mon_vals no callback set 2024-03-25T18:25:07.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.096+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f1324005030 con 0x7f132c0b4e70 2024-03-25T18:25:07.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.096+0000 7f13308d6640 10 monclient: get_auth_request con 0x7f131006e000 auth_method 0 2024-03-25T18:25:07.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.097+0000 7f13297fa640 10 client.4683.objecter ms_handle_connect 0x7f131006e000 2024-03-25T18:25:07.104 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7f13240548a0 con 0x7f132c0b4e70 2024-03-25T18:25:07.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 10 client.4683.objecter ms_dispatch 0x7f132c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:07.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 3 client.4683.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:07.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 3 client.4683.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:07.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 20 client.4683.objecter dump_active .. 0 homeless 2024-03-25T18:25:07.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f13240503b0 con 0x7f132c0b4e70 2024-03-25T18:25:07.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:07.105 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.103+0000 7f13297fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:07.358 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.356+0000 7f1332b3a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:07.358 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.356+0000 7f1332b3a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:07.358 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.356+0000 7f1332b3a640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:07.358 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.356+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f132c13b450 con 0x7f132c0b4e70 2024-03-25T18:25:07.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.383+0000 7f13297fa640 1 -- 172.21.15.7:0/2185903271 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7f13240522e0 con 0x7f132c0b4e70 2024-03-25T18:25:07.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.383+0000 7f13297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:07.385 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.383+0000 7f13297fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:07.385 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:25:07.385 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:07.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 10 librados: watch_flush enter 2024-03-25T18:25:07.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 10 librados: watch_flush exit 2024-03-25T18:25:07.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 20 client.4683.objecter shutdown clearing up homeless session... 2024-03-25T18:25:07.387 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 10 client.4683.objecter successfully canceled tick 2024-03-25T18:25:07.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 >> v1:172.21.15.7:6800/1717564767 conn(0x7f131006e000 legacy=0x7f13100704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 10 monclient: shutdown 2024-03-25T18:25:07.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:07.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 >> v1:172.21.15.7:6789/0 conn(0x7f132c0b4e70 legacy=0x7f132c142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:07.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 shutdown_connections 2024-03-25T18:25:07.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 >> 172.21.15.7:0/2185903271 conn(0x7f132c0617a0 msgr2=0x7f132c0b6690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:07.388 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.386+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 shutdown_connections 2024-03-25T18:25:07.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.387+0000 7f1332b3a640 1 -- 172.21.15.7:0/2185903271 wait complete. 2024-03-25T18:25:07.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:07.387+0000 7f1332b3a640 1 librados: shutdown 2024-03-25T18:25:07.400 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:14.206 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:14.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.322+0000 7fd1ed463640 10 monclient: get_monmap_and_config 2024-03-25T18:25:14.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.322+0000 7fd1ed463640 10 monclient: build_initial_monmap 2024-03-25T18:25:14.323 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.322+0000 7fd1ed463640 10 monclient: monmap: 2024-03-25T18:25:14.323 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:14.323 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:25:14.323205+0000 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:25:14.323205+0000 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:25:14.324 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:14.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.323+0000 7fd1ed463640 1 Processor -- start 2024-03-25T18:25:14.325 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.323+0000 7fd1ed463640 1 -- start start 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.323+0000 7fd1ed463640 10 monclient: init 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.323+0000 7fd1ed463640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.323+0000 7fd1ed463640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.323+0000 7fd1ed463640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e80b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.323+0000 7fd1ed463640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e80b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.324+0000 7fd1ed463640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e813c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.324+0000 7fd1ed463640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80ab0e0 con 0x7fd1e80b4ff0 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.324+0000 7fd1ed463640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80ab3c0 con 0x7fd1e80b4670 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.324+0000 7fd1ed463640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80a9f30 con 0x7fd1e813c480 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.324+0000 7fd1ed463640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:14.326 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.324+0000 7fd1ed463640 10 monclient(hunting): authenticate will time out at 7956.428991s 2024-03-25T18:25:14.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.333+0000 7fd1e77fe640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fd1e813c480 0x7fd1e814dc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:43774/0 (socket says 172.21.15.7:43774) 2024-03-25T18:25:14.334 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.333+0000 7fd1e77fe640 1 -- 172.21.15.7:0/3813623912 learned_addr learned my addr 172.21.15.7:0/3813623912 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:14.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.347+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434011896 0 0) 0x7fd1e80a9f30 con 0x7fd1e813c480 2024-03-25T18:25:14.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.347+0000 7fd1e67fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.347+0000 7fd1e67fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.347+0000 7fd1e67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.347+0000 7fd1e67fc640 10 monclient(hunting): my global_id is 4718 2024-03-25T18:25:14.348 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.347+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1d4003520 con 0x7fd1e813c480 2024-03-25T18:25:14.358 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3267288460 0 0) 0x7fd1d4003520 con 0x7fd1e813c480 2024-03-25T18:25:14.358 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 >> v1:172.21.15.183:6789/0 conn(0x7fd1e80b4670 legacy=0x7fd1e81425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.358 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 >> v1:172.21.15.7:6789/0 conn(0x7fd1e80b4ff0 legacy=0x7fd1e813bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 1 monclient: found mon.noname-b 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80a9b70 con 0x7fd1e813c480 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:14.359 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1e00019f0 con 0x7fd1e813c480 2024-03-25T18:25:14.360 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1ed463640 5 monclient: authenticate success, global_id 4718 2024-03-25T18:25:14.360 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1ed463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:14.360 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.360 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:14.360 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1e67fc640 10 monclient: dump: 2024-03-25T18:25:14.360 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:14.360 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:14.361 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:14.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.356+0000 7fd1ed463640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:14.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1e0001b50 con 0x7fd1e813c480 2024-03-25T18:25:14.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1e67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:14.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1e67fc640 1 -- 172.21.15.7:0/3813623912 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1e0001fa0 con 0x7fd1e813c480 2024-03-25T18:25:14.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1ed463640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:14.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1ed463640 4 set_mon_vals no callback set 2024-03-25T18:25:14.378 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1e67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1e67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1e67fc640 10 monclient: dump: 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:14.379 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:14.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1ed463640 10 monclient: shutdown 2024-03-25T18:25:14.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1ed463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:14.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.377+0000 7fd1ed463640 1 -- 172.21.15.7:0/3813623912 >> v1:172.21.15.7:6790/0 conn(0x7fd1e813c480 legacy=0x7fd1e814dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.378+0000 7fd1ed463640 1 -- 172.21.15.7:0/3813623912 shutdown_connections 2024-03-25T18:25:14.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.378+0000 7fd1ed463640 1 -- 172.21.15.7:0/3813623912 >> 172.21.15.7:0/3813623912 conn(0x7fd1e80617a0 msgr2=0x7fd1e8061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.378+0000 7fd1ed463640 1 -- 172.21.15.7:0/3813623912 shutdown_connections 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.378+0000 7fd1ed463640 1 -- 172.21.15.7:0/3813623912 wait complete. 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.378+0000 7fd1ed463640 10 monclient: build_initial_monmap 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 10 monclient: monmap: 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:14.382 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 1 librados: starting msgr at 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 1 librados: starting objecter 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 1 Processor -- start 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 1 -- start start 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 1 librados: setting wanted keys 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 1 librados: calling monclient init 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.379+0000 7fd1ed463640 10 monclient: init 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 10 monclient(hunting): picked mon.noname-c con 0x7fd1e80b4670 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 10 monclient(hunting): picked mon.noname-b con 0x7fd1e80b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:14.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 10 monclient(hunting): picked mon.noname-a con 0x7fd1e813c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:14.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e813bc10 con 0x7fd1e813c480 2024-03-25T18:25:14.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80b0d10 con 0x7fd1e80b4ff0 2024-03-25T18:25:14.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1e80af590 con 0x7fd1e80b4670 2024-03-25T18:25:14.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:14.384 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.380+0000 7fd1ed463640 10 monclient(hunting): authenticate will time out at 7956.484833s 2024-03-25T18:25:14.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.388+0000 7fd1dffff640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd1e80b4ff0 0x7fd1e813adf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:39528/0 (socket says 172.21.15.7:39528) 2024-03-25T18:25:14.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.388+0000 7fd1dffff640 1 -- 172.21.15.7:0/1036383535 learned_addr learned my addr 172.21.15.7:0/1036383535 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:14.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.392+0000 7fd1df7fe640 10 client.?.objecter ms_handle_connect 0x7fd1e80b4ff0 2024-03-25T18:25:14.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.392+0000 7fd1df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:14.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.402+0000 7fd1df7fe640 10 client.?.objecter ms_handle_connect 0x7fd1e80b4670 2024-03-25T18:25:14.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.402+0000 7fd1df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:14.411 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.410+0000 7fd1df7fe640 10 client.?.objecter ms_handle_connect 0x7fd1e813c480 2024-03-25T18:25:14.411 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.410+0000 7fd1df7fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.416+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4204377694 0 0) 0x7fd1e80b0d10 con 0x7fd1e80b4ff0 2024-03-25T18:25:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.416+0000 7fd1df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.416+0000 7fd1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.416+0000 7fd1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.416+0000 7fd1df7fe640 10 monclient(hunting): my global_id is 4696 2024-03-25T18:25:14.417 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.416+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1c8004eb0 con 0x7fd1e80b4ff0 2024-03-25T18:25:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.422+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2217459082 0 0) 0x7fd1e80af590 con 0x7fd1e80b4670 2024-03-25T18:25:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.422+0000 7fd1df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.422+0000 7fd1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.423 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.422+0000 7fd1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.422+0000 7fd1df7fe640 10 monclient(hunting): my global_id is 4721 2024-03-25T18:25:14.424 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.422+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e80b0d10 con 0x7fd1e80b4670 2024-03-25T18:25:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.425+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595767452 0 0) 0x7fd1e813bc10 con 0x7fd1e813c480 2024-03-25T18:25:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.425+0000 7fd1df7fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.425+0000 7fd1df7fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.425+0000 7fd1df7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.425+0000 7fd1df7fe640 10 monclient(hunting): my global_id is 4689 2024-03-25T18:25:14.427 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.425+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd1e80af590 con 0x7fd1e813c480 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.432+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2898550723 0 0) 0x7fd1e80af590 con 0x7fd1e813c480 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.432+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e813bc10 con 0x7fd1e813c480 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.432+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1e0001e60 con 0x7fd1e813c480 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.432+0000 7fd1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.432+0000 7fd1df7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.432+0000 7fd1df7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.434 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:14.435 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:14.435 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:14.435 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:14.435 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:14.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.432+0000 7fd1df7fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.435+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 241401333 0 0) 0x7fd1c8004eb0 con 0x7fd1e80b4ff0 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.435+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd1e80af590 con 0x7fd1e80b4ff0 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.435+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1d0001a30 con 0x7fd1e80b4ff0 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.435+0000 7fd1df7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.435+0000 7fd1df7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.435+0000 7fd1df7fe640 10 monclient(hunting): dump: 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:14.437 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:14.438 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:14.438 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 137585001 0 0) 0x7fd1e813bc10 con 0x7fd1e813c480 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 >> v1:172.21.15.7:6790/0 conn(0x7fd1e80b4670 legacy=0x7fd1e80ae640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 >> v1:172.21.15.183:6789/0 conn(0x7fd1e80b4ff0 legacy=0x7fd1e813adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 1 monclient: found mon.a 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:14.452 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1e80af250 con 0x7fd1e813c480 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1df7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 5 monclient: authenticate success, global_id 4689 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 10 monclient: _renew_subs 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1e80afab0 con 0x7fd1e813c480 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 10 client.4689.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 10 monclient: _renew_subs 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.450+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1e80aef30 con 0x7fd1e813c480 2024-03-25T18:25:14.453 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.451+0000 7fd1ed463640 1 librados: init done 2024-03-25T18:25:14.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.452+0000 7fd1ed463640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:14.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.452+0000 7fd1ed463640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:14.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.452+0000 7fd1ed463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:14.454 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.452+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1e806dbd0 con 0x7fd1e813c480 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.459+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1e0001fe0 con 0x7fd1e813c480 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.459+0000 7fd1df7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.459+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd1e0004fb0 con 0x7fd1e813c480 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.459+0000 7fd1df7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.459+0000 7fd1df7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.459+0000 7fd1df7fe640 10 monclient: dump: 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:14.461 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:14.462 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:14.462 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:14.462 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.459+0000 7fd1e5ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:14.481 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.480+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fd1e001af80 con 0x7fd1e813c480 2024-03-25T18:25:14.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.481+0000 7fd1e6ffd640 10 monclient: get_auth_request con 0x7fd1c806ded0 auth_method 0 2024-03-25T18:25:14.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.481+0000 7fd1df7fe640 10 client.4689.objecter ms_handle_connect 0x7fd1c806ded0 2024-03-25T18:25:14.776 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7fd1e0054af0 con 0x7fd1e813c480 2024-03-25T18:25:14.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 10 client.4689.objecter ms_dispatch 0x7fd1e8000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:14.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 3 client.4689.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:14.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 3 client.4689.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:14.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 20 client.4689.objecter dump_active .. 0 homeless 2024-03-25T18:25:14.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd1e0050600 con 0x7fd1e813c480 2024-03-25T18:25:14.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:14.777 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:14.775+0000 7fd1df7fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:15.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.037+0000 7fd1ed463640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:15.038 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.037+0000 7fd1ed463640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:15.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.037+0000 7fd1ed463640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:15.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.037+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd1e80688c0 con 0x7fd1e813c480 2024-03-25T18:25:15.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.142+0000 7fd1df7fe640 1 -- 172.21.15.7:0/1036383535 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7fd1e0052530 con 0x7fd1e813c480 2024-03-25T18:25:15.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.142+0000 7fd1df7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:15.144 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.142+0000 7fd1df7fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:15.144 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:25:15.144 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:15.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.144+0000 7fd1ed463640 10 librados: watch_flush enter 2024-03-25T18:25:15.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.144+0000 7fd1ed463640 10 librados: watch_flush exit 2024-03-25T18:25:15.146 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.144+0000 7fd1ed463640 20 client.4689.objecter shutdown clearing up homeless session... 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.144+0000 7fd1ed463640 10 client.4689.objecter successfully canceled tick 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 >> v1:172.21.15.7:6800/1717564767 conn(0x7fd1c806ded0 legacy=0x7fd1c8070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 10 monclient: shutdown 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 >> v1:172.21.15.7:6789/0 conn(0x7fd1e813c480 legacy=0x7fd1e813b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 shutdown_connections 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 >> 172.21.15.7:0/1036383535 conn(0x7fd1e80617a0 msgr2=0x7fd1e81408a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 shutdown_connections 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 1 -- 172.21.15.7:0/1036383535 wait complete. 2024-03-25T18:25:15.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:15.145+0000 7fd1ed463640 1 librados: shutdown 2024-03-25T18:25:15.158 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:19.907 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:26.544 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:19.917 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:26.554 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:19.944 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:26.581 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:19.973 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:26.599 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:19.975 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:26.612 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:19.978 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:26.604 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:19.980 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:26.617 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:20.010 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:26.636 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:20.022 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:26.659 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:21.965 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:22.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.078+0000 7f589a833640 10 monclient: get_monmap_and_config 2024-03-25T18:25:22.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.078+0000 7f589a833640 10 monclient: build_initial_monmap 2024-03-25T18:25:22.079 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.078+0000 7f589a833640 10 monclient: monmap: 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:25:22.079461+0000 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:25:22.079461+0000 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:25:22.080 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:22.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.079+0000 7f589a833640 1 Processor -- start 2024-03-25T18:25:22.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.079+0000 7f589a833640 1 -- start start 2024-03-25T18:25:22.081 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.079+0000 7f589a833640 10 monclient: init 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 10 monclient(hunting): picked mon.noname-b con 0x7f58940b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 10 monclient(hunting): picked mon.noname-a con 0x7f58940b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 10 monclient(hunting): picked mon.noname-c con 0x7f589413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940ab0e0 con 0x7f58940b4670 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940ab3c0 con 0x7f589413c480 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940a9f30 con 0x7f58940b4ff0 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:22.082 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.080+0000 7f589a833640 10 monclient(hunting): authenticate will time out at 7964.184785s 2024-03-25T18:25:22.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.083+0000 7f588bfff640 1 --1- >> v1:172.21.15.7:6789/0 conn(0x7f58940b4670 0x7f58941425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.7:6789/0 says I am v1:172.21.15.7:49968/0 (socket says 172.21.15.7:49968) 2024-03-25T18:25:22.084 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.083+0000 7f588bfff640 1 -- 172.21.15.7:0/2936831481 learned_addr learned my addr 172.21.15.7:0/2936831481 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:22.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.089+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3818136521 0 0) 0x7f58940ab3c0 con 0x7f589413c480 2024-03-25T18:25:22.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.089+0000 7f58937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.089+0000 7f58937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.089+0000 7f58937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.089+0000 7f58937fe640 10 monclient(hunting): my global_id is 4699 2024-03-25T18:25:22.090 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.089+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5880003520 con 0x7f589413c480 2024-03-25T18:25:22.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.090+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4014000337 0 0) 0x7f58940ab0e0 con 0x7f58940b4670 2024-03-25T18:25:22.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.090+0000 7f58937fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.090+0000 7f58937fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.090+0000 7f58937fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.090+0000 7f58937fe640 10 monclient(hunting): my global_id is 4692 2024-03-25T18:25:22.091 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.090+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58940ab3c0 con 0x7f58940b4670 2024-03-25T18:25:22.098 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3882681922 0 0) 0x7f5880003520 con 0x7f589413c480 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 >> v1:172.21.15.7:6790/0 conn(0x7f58940b4ff0 legacy=0x7f589413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 >> v1:172.21.15.7:6789/0 conn(0x7f58940b4670 legacy=0x7f58941425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 1 monclient: found mon.noname-c 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940a9b70 con 0x7f589413c480 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f588c001a10 con 0x7f589413c480 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f589a833640 5 monclient: authenticate success, global_id 4699 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f589a833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:22.099 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f58937fe640 10 monclient: dump: 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:22.100 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.097+0000 7f589a833640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:22.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f588c001b70 con 0x7f589413c480 2024-03-25T18:25:22.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f58937fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:22.107 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f58937fe640 1 -- 172.21.15.7:0/2936831481 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f588c001f80 con 0x7f589413c480 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f58937fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f58937fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f58937fe640 10 monclient: dump: 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f589a833640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:22.108 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.106+0000 7f589a833640 4 set_mon_vals no callback set 2024-03-25T18:25:22.109 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.107+0000 7f589a833640 10 monclient: shutdown 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.107+0000 7f589a833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.107+0000 7f589a833640 1 -- 172.21.15.7:0/2936831481 >> v1:172.21.15.183:6789/0 conn(0x7f589413c480 legacy=0x7f589414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.107+0000 7f589a833640 1 -- 172.21.15.7:0/2936831481 shutdown_connections 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.107+0000 7f589a833640 1 -- 172.21.15.7:0/2936831481 >> 172.21.15.7:0/2936831481 conn(0x7f58940617a0 msgr2=0x7f5894061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.107+0000 7f589a833640 1 -- 172.21.15.7:0/2936831481 shutdown_connections 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.107+0000 7f589a833640 1 -- 172.21.15.7:0/2936831481 wait complete. 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 10 monclient: build_initial_monmap 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 10 monclient: monmap: 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:25:22.110 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 1 librados: starting msgr at 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 1 librados: starting objecter 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 1 Processor -- start 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 1 -- start start 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 1 librados: setting wanted keys 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 1 librados: calling monclient init 2024-03-25T18:25:22.111 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.108+0000 7f589a833640 10 monclient: init 2024-03-25T18:25:22.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:22.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-25T18:25:22.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 10 monclient(hunting): picked mon.noname-b con 0x7f58940b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:22.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 10 monclient(hunting): picked mon.noname-c con 0x7f58940b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:22.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 10 monclient(hunting): picked mon.noname-a con 0x7f589413c480 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:22.112 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940b0610 con 0x7f589413c480 2024-03-25T18:25:22.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940b0880 con 0x7f58940b4670 2024-03-25T18:25:22.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58940b0b60 con 0x7f58940b4ff0 2024-03-25T18:25:22.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:22.113 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.109+0000 7f589a833640 10 monclient(hunting): authenticate will time out at 7964.214233s 2024-03-25T18:25:22.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.116+0000 7f588bfff640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7f58940b4ff0 0x7f58940aefa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:50534/0 (socket says 172.21.15.7:50534) 2024-03-25T18:25:22.117 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.116+0000 7f588bfff640 1 -- 172.21.15.7:0/4128131364 learned_addr learned my addr 172.21.15.7:0/4128131364 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:22.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.118+0000 7f58917fa640 10 client.?.objecter ms_handle_connect 0x7f58940b4ff0 2024-03-25T18:25:22.119 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.118+0000 7f58917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.119+0000 7f58917fa640 10 client.?.objecter ms_handle_connect 0x7f58940b4670 2024-03-25T18:25:22.120 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.119+0000 7f58917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.128+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 394739579 0 0) 0x7f58940b0880 con 0x7f58940b4670 2024-03-25T18:25:22.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.128+0000 7f58917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.128+0000 7f58917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.128+0000 7f58917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.128+0000 7f58917fa640 10 monclient(hunting): my global_id is 4702 2024-03-25T18:25:22.130 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.128+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5874004eb0 con 0x7f58940b4670 2024-03-25T18:25:22.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.132+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2337890070 0 0) 0x7f58940b0b60 con 0x7f58940b4ff0 2024-03-25T18:25:22.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.132+0000 7f58917fa640 10 monclient(hunting): _negotiate 2024-03-25T18:25:22.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.132+0000 7f58917fa640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:22.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.132+0000 7f58917fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:22.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.132+0000 7f58917fa640 10 monclient(hunting): my global_id is 4727 2024-03-25T18:25:22.133 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.132+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58940b0880 con 0x7f58940b4ff0 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.137+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2614047023 0 0) 0x7f5874004eb0 con 0x7f58940b4670 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.137+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58940b0b60 con 0x7f58940b4670 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.137+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5884001930 con 0x7f58940b4670 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.137+0000 7f58917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.137+0000 7f58917fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.137+0000 7f58917fa640 10 monclient(hunting): dump: 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:22.139 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:22.140 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:22.140 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:22.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.137+0000 7f58917fa640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:22.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.141+0000 7f58917fa640 10 client.?.objecter ms_handle_connect 0x7f589413c480 2024-03-25T18:25:22.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.141+0000 7f58917fa640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:22.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.149+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2033297872 0 0) 0x7f58940b0880 con 0x7f58940b4ff0 2024-03-25T18:25:22.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.149+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5874004eb0 con 0x7f58940b4ff0 2024-03-25T18:25:22.150 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.149+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f587c0019f0 con 0x7f58940b4ff0 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.149+0000 7f58917fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.149+0000 7f58917fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.149+0000 7f58917fa640 10 monclient(hunting): dump: 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:22.151 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:22.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2083384862 0 0) 0x7f58940b0b60 con 0x7f58940b4670 2024-03-25T18:25:22.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 >> v1:172.21.15.7:6790/0 conn(0x7f58940b4ff0 legacy=0x7f58940aefa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 >> v1:172.21.15.7:6789/0 conn(0x7f589413c480 legacy=0x7f58941e2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.157 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 10 monclient: _finish_hunting 0 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 1 monclient: found mon.b 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58940b13f0 con 0x7f58940b4670 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 10 monclient: _finish_auth 0 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 10 monclient: _check_auth_tickets 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f58917fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 5 monclient: authenticate success, global_id 4702 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 10 monclient: _renew_subs 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58940b1910 con 0x7f58940b4670 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 10 client.4702.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 10 monclient: _renew_subs 2024-03-25T18:25:22.158 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:22.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58940af6b0 con 0x7f58940b4670 2024-03-25T18:25:22.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.156+0000 7f589a833640 1 librados: init done 2024-03-25T18:25:22.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.157+0000 7f589a833640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.157+0000 7f589a833640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.157+0000 7f589a833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:22.159 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.157+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58940b1240 con 0x7f58940b4670 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.480+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58840035f0 con 0x7f58940b4670 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.480+0000 7f58917fa640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.480+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7f5884003900 con 0x7f58940b4670 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.480+0000 7f58917fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.480+0000 7f58917fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.480+0000 7f58917fa640 10 monclient: dump: 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.482 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.480+0000 7f5892ffd640 4 set_mon_vals no callback set 2024-03-25T18:25:22.483 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.481+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7f58840198d0 con 0x7f58940b4670 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.482+0000 7f588bfff640 10 monclient: get_auth_request con 0x7f587406ded0 auth_method 0 2024-03-25T18:25:22.484 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.482+0000 7f58917fa640 10 client.4702.objecter ms_handle_connect 0x7f587406ded0 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7f58840535e0 con 0x7f58940b4670 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 10 client.4702.objecter ms_dispatch 0x7f5894000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 3 client.4702.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 3 client.4702.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 20 client.4702.objecter dump_active .. 0 homeless 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f588404ef20 con 0x7f58940b4670 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:22.489 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.487+0000 7f58917fa640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:22.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.714+0000 7f589a833640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:22.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.714+0000 7f589a833640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:22.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.714+0000 7f589a833640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:22.716 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.714+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f58940b1580 con 0x7f58940b4670 2024-03-25T18:25:22.756 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.755+0000 7f58917fa640 1 -- 172.21.15.7:0/4128131364 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7f5884051020 con 0x7f58940b4670 2024-03-25T18:25:22.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.755+0000 7f58917fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:22.757 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.755+0000 7f58917fa640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:22.757 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:25:22.757 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:22.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.758+0000 7f589a833640 10 librados: watch_flush enter 2024-03-25T18:25:22.760 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.758+0000 7f589a833640 10 librados: watch_flush exit 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.758+0000 7f589a833640 20 client.4702.objecter shutdown clearing up homeless session... 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.758+0000 7f589a833640 10 client.4702.objecter successfully canceled tick 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 >> v1:172.21.15.7:6800/1717564767 conn(0x7f587406ded0 legacy=0x7f5874070390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 10 monclient: shutdown 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 >> v1:172.21.15.183:6789/0 conn(0x7f58940b4670 legacy=0x7f58940ae770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 shutdown_connections 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 >> 172.21.15.7:0/4128131364 conn(0x7f58940617a0 msgr2=0x7f5894145670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 shutdown_connections 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 1 -- 172.21.15.7:0/4128131364 wait complete. 2024-03-25T18:25:22.761 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:22.759+0000 7f589a833640 1 librados: shutdown 2024-03-25T18:25:22.771 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:23.759 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:30.396 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.763 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:30.400 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.772 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:30.408 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.773 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:30.399 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.776 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:30.402 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.778 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:30.415 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.786 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:30.423 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.795 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:30.421 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:23.849 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:30.486 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:29.578 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:29.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.690+0000 7fd59a8c8640 10 monclient: get_monmap_and_config 2024-03-25T18:25:29.691 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.690+0000 7fd59a8c8640 10 monclient: build_initial_monmap 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.690+0000 7fd59a8c8640 10 monclient: monmap: 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:25:29.691506+0000 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:25:29.691506+0000 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:25:29.692 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:29.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.691+0000 7fd59a8c8640 1 Processor -- start 2024-03-25T18:25:29.693 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.691+0000 7fd59a8c8640 1 -- start start 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.691+0000 7fd59a8c8640 10 monclient: init 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5940b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd5940b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 10 monclient(hunting): picked mon.noname-b con 0x7fd59413c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940ab0e0 con 0x7fd5940b4ff0 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940ab3c0 con 0x7fd5940b4670 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940a9f30 con 0x7fd59413c480 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:29.694 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd59a8c8640 10 monclient(hunting): authenticate will time out at 7971.796913s 2024-03-25T18:25:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd5937fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd5940b4670 0x7fd5941425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:43922/0 (socket says 172.21.15.7:43922) 2024-03-25T18:25:29.695 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.692+0000 7fd5937fe640 1 -- 172.21.15.7:0/3550720913 learned_addr learned my addr 172.21.15.7:0/3550720913 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:29.768 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.767+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 852816063 0 0) 0x7fd5940a9f30 con 0x7fd59413c480 2024-03-25T18:25:29.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.767+0000 7fd592ffd640 10 monclient(hunting): _negotiate 2024-03-25T18:25:29.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.767+0000 7fd592ffd640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:29.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.767+0000 7fd592ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:29.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.767+0000 7fd592ffd640 10 monclient(hunting): my global_id is 4730 2024-03-25T18:25:29.769 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.767+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd57c003520 con 0x7fd59413c480 2024-03-25T18:25:29.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3442080636 0 0) 0x7fd57c003520 con 0x7fd59413c480 2024-03-25T18:25:29.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 >> v1:172.21.15.183:6789/0 conn(0x7fd5940b4670 legacy=0x7fd5941425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:29.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 >> v1:172.21.15.7:6789/0 conn(0x7fd5940b4ff0 legacy=0x7fd59413bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:29.773 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 10 monclient: _finish_hunting 0 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 1 monclient: found mon.noname-b 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.7:6790/0 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 --> v1:172.21.15.7:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5940a9b70 con 0x7fd59413c480 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 10 monclient: _finish_auth 0 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 10 monclient: _check_auth_tickets 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd58c001a10 con 0x7fd59413c480 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:29.774 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd592ffd640 10 monclient: dump: 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd59a8c8640 5 monclient: authenticate success, global_id 4730 2024-03-25T18:25:29.775 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.772+0000 7fd59a8c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:29.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 <== mon.2 v1:172.21.15.7:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd58c001b70 con 0x7fd59413c480 2024-03-25T18:25:29.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd592ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:29.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd592ffd640 1 -- 172.21.15.7:0/3550720913 <== mon.2 v1:172.21.15.7:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd58c001f80 con 0x7fd59413c480 2024-03-25T18:25:29.800 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd592ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd592ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd592ffd640 10 monclient: dump: 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd59a8c8640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:29.801 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.799+0000 7fd59a8c8640 4 set_mon_vals no callback set 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 10 monclient: shutdown 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 1 -- 172.21.15.7:0/3550720913 >> v1:172.21.15.7:6790/0 conn(0x7fd59413c480 legacy=0x7fd59414dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 1 -- 172.21.15.7:0/3550720913 shutdown_connections 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 1 -- 172.21.15.7:0/3550720913 >> 172.21.15.7:0/3550720913 conn(0x7fd5940617a0 msgr2=0x7fd594061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 1 -- 172.21.15.7:0/3550720913 shutdown_connections 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 1 -- 172.21.15.7:0/3550720913 wait complete. 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 10 monclient: build_initial_monmap 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.800+0000 7fd59a8c8640 10 monclient: monmap: 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:25:29.804 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.801+0000 7fd59a8c8640 1 librados: starting msgr at 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.801+0000 7fd59a8c8640 1 librados: starting objecter 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.801+0000 7fd59a8c8640 1 Processor -- start 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.801+0000 7fd59a8c8640 1 -- start start 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.801+0000 7fd59a8c8640 1 librados: setting wanted keys 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.801+0000 7fd59a8c8640 1 librados: calling monclient init 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.801+0000 7fd59a8c8640 10 monclient: init 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 10 monclient(hunting): picked mon.noname-a con 0x7fd5940b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:29.805 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 10 monclient(hunting): picked mon.noname-c con 0x7fd5940b4ff0 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:29.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 10 monclient(hunting): picked mon.noname-b con 0x7fd59413c480 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:29.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940b0ba0 con 0x7fd5940b4670 2024-03-25T18:25:29.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940b0e80 con 0x7fd59413c480 2024-03-25T18:25:29.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5940afba0 con 0x7fd5940b4ff0 2024-03-25T18:25:29.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:29.806 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.802+0000 7fd59a8c8640 10 monclient(hunting): authenticate will time out at 7971.906925s 2024-03-25T18:25:29.816 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.815+0000 7fd598e65640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fd59413c480 0x7fd5940aef30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:43976/0 (socket says 172.21.15.7:43976) 2024-03-25T18:25:29.816 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.815+0000 7fd598e65640 1 -- 172.21.15.7:0/4109280906 learned_addr learned my addr 172.21.15.7:0/4109280906 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:29.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.819+0000 7fd590ff9640 10 client.?.objecter ms_handle_connect 0x7fd59413c480 2024-03-25T18:25:29.820 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.819+0000 7fd590ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:29.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.821+0000 7fd590ff9640 10 client.?.objecter ms_handle_connect 0x7fd5940b4ff0 2024-03-25T18:25:29.822 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.821+0000 7fd590ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:29.832 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.831+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366967218 0 0) 0x7fd5940afba0 con 0x7fd5940b4ff0 2024-03-25T18:25:29.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.831+0000 7fd590ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:29.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.831+0000 7fd590ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:29.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.831+0000 7fd590ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:29.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.831+0000 7fd590ff9640 10 monclient(hunting): my global_id is 4733 2024-03-25T18:25:29.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.831+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd580004eb0 con 0x7fd5940b4ff0 2024-03-25T18:25:29.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.832+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477853058 0 0) 0x7fd5940b0e80 con 0x7fd59413c480 2024-03-25T18:25:29.833 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.832+0000 7fd590ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:29.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.832+0000 7fd590ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:29.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.832+0000 7fd590ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:29.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.832+0000 7fd590ff9640 10 monclient(hunting): my global_id is 4708 2024-03-25T18:25:29.834 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.832+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5940afba0 con 0x7fd59413c480 2024-03-25T18:25:29.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.842+0000 7fd590ff9640 10 client.?.objecter ms_handle_connect 0x7fd5940b4670 2024-03-25T18:25:29.843 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.842+0000 7fd590ff9640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:29.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.848+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2328376155 0 0) 0x7fd5940afba0 con 0x7fd59413c480 2024-03-25T18:25:29.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.848+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5940b0e80 con 0x7fd59413c480 2024-03-25T18:25:29.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.848+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd58c001f00 con 0x7fd59413c480 2024-03-25T18:25:29.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.848+0000 7fd590ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:29.849 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.848+0000 7fd590ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.848+0000 7fd590ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:29.850 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.848+0000 7fd590ff9640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:29.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1389508537 0 0) 0x7fd5940b0ba0 con 0x7fd5940b4670 2024-03-25T18:25:29.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 10 monclient(hunting): _negotiate 2024-03-25T18:25:29.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:29.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:29.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 10 monclient(hunting): my global_id is 4701 2024-03-25T18:25:29.852 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd5940afba0 con 0x7fd5940b4670 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 277268792 0 0) 0x7fd580004eb0 con 0x7fd5940b4ff0 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd5940b0ba0 con 0x7fd5940b4ff0 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd5880018a0 con 0x7fd5940b4ff0 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.851+0000 7fd590ff9640 10 monclient(hunting): dump: 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:29.853 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:29.854 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:29.854 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:29.854 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:29.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2624134396 0 0) 0x7fd5940b0e80 con 0x7fd59413c480 2024-03-25T18:25:29.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 >> v1:172.21.15.7:6790/0 conn(0x7fd5940b4ff0 legacy=0x7fd5940ae820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:29.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 >> v1:172.21.15.7:6789/0 conn(0x7fd5940b4670 legacy=0x7fd5941db720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:29.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 10 monclient: _finish_hunting 0 2024-03-25T18:25:29.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 1 monclient: found mon.b 2024-03-25T18:25:29.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:29.863 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5940af640 con 0x7fd59413c480 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 10 monclient: _finish_auth 0 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 10 monclient: _check_auth_tickets 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.861+0000 7fd590ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 5 monclient: authenticate success, global_id 4708 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 10 monclient: _renew_subs 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:29.864 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5940af8f0 con 0x7fd59413c480 2024-03-25T18:25:29.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 10 client.4708.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:29.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 10 monclient: _renew_subs 2024-03-25T18:25:29.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:29.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59414ec20 con 0x7fd59413c480 2024-03-25T18:25:29.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.862+0000 7fd59a8c8640 1 librados: init done 2024-03-25T18:25:29.865 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.863+0000 7fd59a8c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:29.866 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.863+0000 7fd59a8c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:29.866 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.863+0000 7fd59a8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:29.866 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:29.863+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59406dbd0 con 0x7fd59413c480 2024-03-25T18:25:30.140 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.139+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd58c003c20 con 0x7fd59413c480 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.139+0000 7fd590ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.139+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fd58c003fb0 con 0x7fd59413c480 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.139+0000 7fd590ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.139+0000 7fd590ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.139+0000 7fd590ff9640 10 monclient: dump: 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:30.141 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:30.142 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:30.142 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:30.142 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:30.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.139+0000 7fd5927fc640 4 set_mon_vals no callback set 2024-03-25T18:25:30.142 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.140+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fd58c0041d0 con 0x7fd59413c480 2024-03-25T18:25:30.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.141+0000 7fd593fff640 10 monclient: get_auth_request con 0x7fd58006dfd0 auth_method 0 2024-03-25T18:25:30.143 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.141+0000 7fd590ff9640 10 client.4708.objecter ms_handle_connect 0x7fd58006dfd0 2024-03-25T18:25:30.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.145+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7fd58c054b30 con 0x7fd59413c480 2024-03-25T18:25:30.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.145+0000 7fd590ff9640 10 client.4708.objecter ms_dispatch 0x7fd594000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:30.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.145+0000 7fd590ff9640 3 client.4708.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:30.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.145+0000 7fd590ff9640 3 client.4708.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:30.147 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.145+0000 7fd590ff9640 20 client.4708.objecter dump_active .. 0 homeless 2024-03-25T18:25:30.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.147+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd58c050640 con 0x7fd59413c480 2024-03-25T18:25:30.148 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.147+0000 7fd590ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:30.149 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.147+0000 7fd590ff9640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:30.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.395+0000 7fd59a8c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:30.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.395+0000 7fd59a8c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:30.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.395+0000 7fd59a8c8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:30.397 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.395+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd594141e00 con 0x7fd59413c480 2024-03-25T18:25:30.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.425+0000 7fd590ff9640 1 -- 172.21.15.7:0/4109280906 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7fd58c052570 con 0x7fd59413c480 2024-03-25T18:25:30.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.425+0000 7fd590ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:30.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.425+0000 7fd590ff9640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:30.426 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:25:30.426 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 10 librados: watch_flush enter 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 10 librados: watch_flush exit 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 20 client.4708.objecter shutdown clearing up homeless session... 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 10 client.4708.objecter successfully canceled tick 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 >> v1:172.21.15.7:6800/1717564767 conn(0x7fd58006dfd0 legacy=0x7fd580070490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 10 monclient: shutdown 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 >> v1:172.21.15.183:6789/0 conn(0x7fd59413c480 legacy=0x7fd5940aef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:30.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 shutdown_connections 2024-03-25T18:25:30.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 >> 172.21.15.7:0/4109280906 conn(0x7fd5940617a0 msgr2=0x7fd5940b6d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:30.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 shutdown_connections 2024-03-25T18:25:30.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 1 -- 172.21.15.7:0/4109280906 wait complete. 2024-03-25T18:25:30.430 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:30.427+0000 7fd59a8c8640 1 librados: shutdown 2024-03-25T18:25:30.440 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:37.247 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:37.362 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.361+0000 7fe2a380f640 10 monclient: get_monmap_and_config 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.361+0000 7fe2a380f640 10 monclient: build_initial_monmap 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.361+0000 7fe2a380f640 10 monclient: monmap: 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:25:37.362140+0000 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:25:37.362140+0000 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:37.363 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.361+0000 7fe2a380f640 1 Processor -- start 2024-03-25T18:25:37.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 1 -- start start 2024-03-25T18:25:37.364 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient: init 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient(hunting): picked mon.noname-a con 0x7fe29c0b4ff0 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient(hunting): picked mon.noname-c con 0x7fe29c0b4670 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient(hunting): picked mon.noname-b con 0x7fe29c13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0ab0e0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0ab3c0 con 0x7fe29c0b4670 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0a9f30 con 0x7fe29c13c480 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:37.365 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.362+0000 7fe2a380f640 10 monclient(hunting): authenticate will time out at 7979.467502s 2024-03-25T18:25:37.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.367+0000 7fe2a0daa640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7fe29c0b4670 0x7fe29c1425c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:35138/0 (socket says 172.21.15.7:35138) 2024-03-25T18:25:37.368 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.367+0000 7fe2a0daa640 1 -- 172.21.15.7:0/1215171289 learned_addr learned my addr 172.21.15.7:0/1215171289 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:37.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.381+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2779248395 0 0) 0x7fe29c0ab3c0 con 0x7fe29c0b4670 2024-03-25T18:25:37.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.381+0000 7fe293fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.381+0000 7fe293fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.381+0000 7fe293fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.381+0000 7fe293fff640 10 monclient(hunting): my global_id is 4711 2024-03-25T18:25:37.383 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.381+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe288003520 con 0x7fe29c0b4670 2024-03-25T18:25:37.389 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.388+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2319442267 0 0) 0x7fe29c0a9f30 con 0x7fe29c13c480 2024-03-25T18:25:37.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.388+0000 7fe293fff640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.388+0000 7fe293fff640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.388+0000 7fe293fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.388+0000 7fe293fff640 10 monclient(hunting): my global_id is 4736 2024-03-25T18:25:37.390 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.388+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c0ab3c0 con 0x7fe29c13c480 2024-03-25T18:25:37.393 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1006471670 0 0) 0x7fe288003520 con 0x7fe29c0b4670 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 >> v1:172.21.15.7:6790/0 conn(0x7fe29c13c480 legacy=0x7fe29c14dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 >> v1:172.21.15.7:6789/0 conn(0x7fe29c0b4ff0 legacy=0x7fe29c13bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 10 monclient: _finish_hunting 0 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 1 monclient: found mon.noname-c 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c0a9b70 con 0x7fe29c0b4670 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 10 monclient: _finish_auth 0 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 10 monclient: _check_auth_tickets 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:37.394 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe28c001690 con 0x7fe29c0b4670 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe2a380f640 5 monclient: authenticate success, global_id 4711 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe2a380f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe293fff640 10 monclient: dump: 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:37.395 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.392+0000 7fe2a380f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:37.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe28c0017f0 con 0x7fe29c0b4670 2024-03-25T18:25:37.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe293fff640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:37.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe293fff640 1 -- 172.21.15.7:0/1215171289 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe28c001c00 con 0x7fe29c0b4670 2024-03-25T18:25:37.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe293fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.398 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe293fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe293fff640 10 monclient: dump: 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe2a380f640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:37.399 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.397+0000 7fe2a380f640 4 set_mon_vals no callback set 2024-03-25T18:25:37.400 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.398+0000 7fe2a380f640 10 monclient: shutdown 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.398+0000 7fe2a380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.398+0000 7fe2a380f640 1 -- 172.21.15.7:0/1215171289 >> v1:172.21.15.183:6789/0 conn(0x7fe29c0b4670 legacy=0x7fe29c1425c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.398+0000 7fe2a380f640 1 -- 172.21.15.7:0/1215171289 shutdown_connections 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.398+0000 7fe2a380f640 1 -- 172.21.15.7:0/1215171289 >> 172.21.15.7:0/1215171289 conn(0x7fe29c0617a0 msgr2=0x7fe29c061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.398+0000 7fe2a380f640 1 -- 172.21.15.7:0/1215171289 shutdown_connections 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.398+0000 7fe2a380f640 1 -- 172.21.15.7:0/1215171289 wait complete. 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 10 monclient: build_initial_monmap 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 10 monclient: monmap: 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:37.401 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 1 librados: starting msgr at 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 1 librados: starting objecter 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 1 Processor -- start 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 1 -- start start 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 1 librados: setting wanted keys 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 1 librados: calling monclient init 2024-03-25T18:25:37.402 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.399+0000 7fe2a380f640 10 monclient: init 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 10 monclient(hunting): picked mon.noname-a con 0x7fe29c0b4670 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 10 monclient(hunting): picked mon.noname-b con 0x7fe29c0b4ff0 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 10 monclient(hunting): picked mon.noname-c con 0x7fe29c13c480 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0b05a0 con 0x7fe29c0b4670 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0b0880 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe29c0b02c0 con 0x7fe29c13c480 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:37.403 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.400+0000 7fe2a380f640 10 monclient(hunting): authenticate will time out at 7979.504990s 2024-03-25T18:25:37.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.411+0000 7fe2a1dac640 1 --1- >> v1:172.21.15.7:6790/0 conn(0x7fe29c13c480 0x7fe29c0aed10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.7:6790/0 says I am v1:172.21.15.7:57278/0 (socket says 172.21.15.7:57278) 2024-03-25T18:25:37.412 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.411+0000 7fe2a1dac640 1 -- 172.21.15.7:0/1685202491 learned_addr learned my addr 172.21.15.7:0/1685202491 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:37.413 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.412+0000 7fe291ffb640 10 client.?.objecter ms_handle_connect 0x7fe29c13c480 2024-03-25T18:25:37.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.412+0000 7fe291ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:37.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.412+0000 7fe291ffb640 10 client.?.objecter ms_handle_connect 0x7fe29c0b4ff0 2024-03-25T18:25:37.414 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.412+0000 7fe291ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:37.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.417+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2736429299 0 0) 0x7fe29c0b0880 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.417+0000 7fe291ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.417+0000 7fe291ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.417+0000 7fe291ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.418 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.417+0000 7fe291ffb640 10 monclient(hunting): my global_id is 4714 2024-03-25T18:25:37.419 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.417+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe280004eb0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.420+0000 7fe291ffb640 10 client.?.objecter ms_handle_connect 0x7fe29c0b4670 2024-03-25T18:25:37.421 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.420+0000 7fe291ffb640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:37.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2582048655 0 0) 0x7fe29c0b02c0 con 0x7fe29c13c480 2024-03-25T18:25:37.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.425 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 10 monclient(hunting): my global_id is 4739 2024-03-25T18:25:37.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c0b0880 con 0x7fe29c13c480 2024-03-25T18:25:37.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 62921854 0 0) 0x7fe280004eb0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.426 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.424+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.183:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe29c0b02c0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.427+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe28c001bc0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.427+0000 7fe291ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.427+0000 7fe291ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-25T18:25:37.428 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.427+0000 7fe291ffb640 10 monclient(hunting): dump: 2024-03-25T18:25:37.428 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:37.429 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.427+0000 7fe291ffb640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:37.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.2 v1:172.21.15.7:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 604222782 0 0) 0x7fe29c0b0880 con 0x7fe29c13c480 2024-03-25T18:25:37.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.7:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe280004eb0 con 0x7fe29c13c480 2024-03-25T18:25:37.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.2 v1:172.21.15.7:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe294001c10 con 0x7fe29c13c480 2024-03-25T18:25:37.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-25T18:25:37.435 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 10 monclient(hunting): dump: 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2391894084 0 0) 0x7fe29c0b05a0 con 0x7fe29c0b4670 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 10 monclient(hunting): _negotiate 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 10 monclient(hunting): my global_id is 4707 2024-03-25T18:25:37.436 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.434+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe29c0b0880 con 0x7fe29c0b4670 2024-03-25T18:25:37.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1672034255 0 0) 0x7fe29c0b02c0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 >> v1:172.21.15.7:6790/0 conn(0x7fe29c13c480 legacy=0x7fe29c0aed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 >> v1:172.21.15.7:6789/0 conn(0x7fe29c0b4670 legacy=0x7fe29c0b0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:37.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 10 monclient: _finish_hunting 0 2024-03-25T18:25:37.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 1 monclient: found mon.b 2024-03-25T18:25:37.443 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe29c0b13b0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 10 monclient: _finish_auth 0 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 10 monclient: _check_auth_tickets 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe291ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 5 monclient: authenticate success, global_id 4714 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 10 monclient: _renew_subs 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.183:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe29c0b1660 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 10 client.4714.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 10 monclient: _renew_subs 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.183:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe29c1e3780 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.444 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.442+0000 7fe2a380f640 1 librados: init done 2024-03-25T18:25:37.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.443+0000 7fe2a380f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:37.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.443+0000 7fe2a380f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:37.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.443+0000 7fe2a380f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:37.445 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.443+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe29c076320 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.666+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe28c004a90 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.666+0000 7fe291ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:37.668 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.667+0000 7fe2937fe640 4 set_mon_vals no callback set 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.707+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7fe28c004da0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.707+0000 7fe291ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.707+0000 7fe291ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.707+0000 7fe291ffb640 10 monclient: dump: 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.708 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:37.709 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:37.709 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:37.709 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:37.709 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:37.709 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:37.709 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:37.717 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.716+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7fe28c01ad70 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.718 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.717+0000 7fe2a15ab640 10 monclient: get_auth_request con 0x7fe28006df30 auth_method 0 2024-03-25T18:25:37.719 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.718+0000 7fe291ffb640 10 client.4714.objecter ms_handle_connect 0x7fe28006df30 2024-03-25T18:25:37.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.719+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7fe28c0549c0 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.719+0000 7fe291ffb640 10 client.4714.objecter ms_dispatch 0x7fe29c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:37.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.719+0000 7fe291ffb640 3 client.4714.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:37.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.719+0000 7fe291ffb640 3 client.4714.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:37.721 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.719+0000 7fe291ffb640 20 client.4714.objecter dump_active .. 0 homeless 2024-03-25T18:25:37.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.723+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe28c050300 con 0x7fe29c0b4ff0 2024-03-25T18:25:37.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.723+0000 7fe291ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:37.724 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.723+0000 7fe291ffb640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:37.993 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.992+0000 7fe2a380f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:37.994 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.992+0000 7fe2a380f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:37.994 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.992+0000 7fe2a380f640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.183:6789/0 2024-03-25T18:25:37.994 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:37.992+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 --> v1:172.21.15.183:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe29c06dbd0 con 0x7fe29c0b4ff0 2024-03-25T18:25:38.010 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.008+0000 7fe291ffb640 1 -- 172.21.15.7:0/1685202491 <== mon.1 v1:172.21.15.183:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7fe28c052400 con 0x7fe29c0b4ff0 2024-03-25T18:25:38.010 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.008+0000 7fe291ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:38.010 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.008+0000 7fe291ffb640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:38.010 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:25:38.010 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:38.013 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 10 librados: watch_flush enter 2024-03-25T18:25:38.013 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 10 librados: watch_flush exit 2024-03-25T18:25:38.013 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 20 client.4714.objecter shutdown clearing up homeless session... 2024-03-25T18:25:38.013 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 10 client.4714.objecter successfully canceled tick 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 >> v1:172.21.15.7:6800/1717564767 conn(0x7fe28006df30 legacy=0x7fe2800703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 10 monclient: shutdown 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 >> v1:172.21.15.183:6789/0 conn(0x7fe29c0b4ff0 legacy=0x7fe29c0ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 shutdown_connections 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.011+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 >> 172.21.15.7:0/1685202491 conn(0x7fe29c0617a0 msgr2=0x7fe29c0b6e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.012+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 shutdown_connections 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.012+0000 7fe2a380f640 1 -- 172.21.15.7:0/1685202491 wait complete. 2024-03-25T18:25:38.014 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:38.012+0000 7fe2a380f640 1 librados: shutdown 2024-03-25T18:25:38.024 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:39.907 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:46.544 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:39.967 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:46.603 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:39.994 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:11:46.631 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.007 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:46.633 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.018 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:46.644 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.020 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:46.657 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.033 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:46.670 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.044 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:11:46.670 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:40.066 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:11:46.702 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:44.831 DEBUG:teuthology.orchestra.run.smithi007:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-25T18:25:44.940 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.939+0000 7ff4712f6640 10 monclient: get_monmap_and_config 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.939+0000 7ff4712f6640 10 monclient: build_initial_monmap 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.939+0000 7ff4712f6640 10 monclient: monmap: 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:25:44.940641+0000 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:25:44.940641+0000 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-c 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-b 2024-03-25T18:25:44.941 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:44.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.940+0000 7ff4712f6640 1 Processor -- start 2024-03-25T18:25:44.942 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.940+0000 7ff4712f6640 1 -- start start 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.940+0000 7ff4712f6640 10 monclient: init 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff46c06f460 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff46c06f840 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff46c0728f0 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c071a40 con 0x7ff46c06f460 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c071db0 con 0x7ff46c0728f0 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c070650 con 0x7ff46c06f840 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:44.943 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.941+0000 7ff4712f6640 10 monclient(hunting): authenticate will time out at 7987.045888s 2024-03-25T18:25:44.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.950+0000 7ff46b7fe640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff46c0728f0 0x7ff46c14a8f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:33276/0 (socket says 172.21.15.7:33276) 2024-03-25T18:25:44.951 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.950+0000 7ff46b7fe640 1 -- 172.21.15.7:0/151864513 learned_addr learned my addr 172.21.15.7:0/151864513 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:44.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.972+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3201463795 0 0) 0x7ff46c071db0 con 0x7ff46c0728f0 2024-03-25T18:25:44.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.972+0000 7ff46a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:44.973 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.972+0000 7ff46a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:44.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.972+0000 7ff46a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:44.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.972+0000 7ff46a7fc640 10 monclient(hunting): my global_id is 4717 2024-03-25T18:25:44.974 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.972+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff458005730 con 0x7ff46c0728f0 2024-03-25T18:25:44.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.977+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 <== mon.2 v1:172.21.15.7:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2018356253 0 0) 0x7ff46c070650 con 0x7ff46c06f840 2024-03-25T18:25:44.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.977+0000 7ff46a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:44.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.977+0000 7ff46a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:44.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.977+0000 7ff46a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:44.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.977+0000 7ff46a7fc640 10 monclient(hunting): my global_id is 4742 2024-03-25T18:25:44.978 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.977+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 --> v1:172.21.15.7:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff46c071db0 con 0x7ff46c06f840 2024-03-25T18:25:44.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.988+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225639630 0 0) 0x7ff46c071a40 con 0x7ff46c06f460 2024-03-25T18:25:44.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.988+0000 7ff46a7fc640 10 monclient(hunting): _negotiate 2024-03-25T18:25:44.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.988+0000 7ff46a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:44.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.988+0000 7ff46a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:44.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.988+0000 7ff46a7fc640 10 monclient(hunting): my global_id is 4710 2024-03-25T18:25:44.989 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.988+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff46c070650 con 0x7ff46c06f460 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 <== mon.1 v1:172.21.15.183:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 513049347 0 0) 0x7ff458005730 con 0x7ff46c0728f0 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 >> v1:172.21.15.7:6790/0 conn(0x7ff46c06f840 legacy=0x7ff46c0721e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 >> v1:172.21.15.7:6789/0 conn(0x7ff46c06f460 legacy=0x7ff46c142cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 10 monclient: _finish_hunting 0 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 1 monclient: found mon.noname-c 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.183:6789/0 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 --> v1:172.21.15.183:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c070200 con 0x7ff46c0728f0 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 10 monclient: _finish_auth 0 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 10 monclient: _check_auth_tickets 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 <== mon.1 v1:172.21.15.183:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff464001a10 con 0x7ff46c0728f0 2024-03-25T18:25:44.995 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff46a7fc640 10 monclient: dump: 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff4712f6640 5 monclient: authenticate success, global_id 4717 2024-03-25T18:25:44.996 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:44.993+0000 7ff4712f6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-25T18:25:45.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 <== mon.1 v1:172.21.15.183:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff464001b70 con 0x7ff46c0728f0 2024-03-25T18:25:45.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff46a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:45.003 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff46a7fc640 1 -- 172.21.15.7:0/151864513 <== mon.1 v1:172.21.15.183:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff464001f00 con 0x7ff46c0728f0 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff46a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff46a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff46a7fc640 10 monclient: dump: 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:45.004 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff4712f6640 10 monclient: get_monmap_and_config success 2024-03-25T18:25:45.005 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.002+0000 7ff4712f6640 4 set_mon_vals no callback set 2024-03-25T18:25:45.006 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 10 monclient: shutdown 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 1 -- 172.21.15.7:0/151864513 >> v1:172.21.15.183:6789/0 conn(0x7ff46c0728f0 legacy=0x7ff46c14a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 1 -- 172.21.15.7:0/151864513 shutdown_connections 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 1 -- 172.21.15.7:0/151864513 >> 172.21.15.7:0/151864513 conn(0x7ff46c0b4bd0 msgr2=0x7ff46c0b4fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 1 -- 172.21.15.7:0/151864513 shutdown_connections 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 1 -- 172.21.15.7:0/151864513 wait complete. 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 10 monclient: build_initial_monmap 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.003+0000 7ff4712f6640 10 monclient: monmap: 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:epoch 0 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:last_changed 0.000000 2024-03-25T18:25:45.007 INFO:teuthology.misc.health.smithi007.stderr:created 0.000000 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 0 (unknown) 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 1 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.noname-a 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.noname-b 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.noname-c 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 1 librados: starting msgr at 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 1 librados: starting objecter 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 1 Processor -- start 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 1 -- start start 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 1 librados: setting wanted keys 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 1 librados: calling monclient init 2024-03-25T18:25:45.008 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 10 monclient: init 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 10 monclient: _reopen_session rank -1 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.004+0000 7ff4712f6640 10 monclient(hunting): picked mon.noname-b con 0x7ff46c06f460 addr v1:172.21.15.183:6789/0 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.005+0000 7ff4712f6640 10 monclient(hunting): picked mon.noname-a con 0x7ff46c06f840 addr v1:172.21.15.7:6789/0 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.005+0000 7ff4712f6640 10 monclient(hunting): picked mon.noname-c con 0x7ff46c0728f0 addr v1:172.21.15.7:6790/0 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.005+0000 7ff4712f6640 1 -- --> v1:172.21.15.7:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c1d8340 con 0x7ff46c06f840 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.005+0000 7ff4712f6640 1 -- --> v1:172.21.15.183:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c1d87d0 con 0x7ff46c06f460 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.005+0000 7ff4712f6640 1 -- --> v1:172.21.15.7:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff46c1360c0 con 0x7ff46c0728f0 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.005+0000 7ff4712f6640 10 monclient(hunting): _renew_subs 2024-03-25T18:25:45.009 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.005+0000 7ff4712f6640 10 monclient(hunting): authenticate will time out at 7987.109783s 2024-03-25T18:25:45.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.018+0000 7ff46affd640 1 --1- >> v1:172.21.15.183:6789/0 conn(0x7ff46c06f460 0x7ff46c1d79a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.183:6789/0 says I am v1:172.21.15.7:33308/0 (socket says 172.21.15.7:33308) 2024-03-25T18:25:45.019 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.018+0000 7ff46affd640 1 -- 172.21.15.7:0/3119193430 learned_addr learned my addr 172.21.15.7:0/3119193430 (peer_addr_for_me v1:172.21.15.7:0/0) 2024-03-25T18:25:45.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.019+0000 7ff4637fe640 10 client.?.objecter ms_handle_connect 0x7ff46c06f460 2024-03-25T18:25:45.020 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.019+0000 7ff4637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:45.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.020+0000 7ff4637fe640 10 client.?.objecter ms_handle_connect 0x7ff46c06f840 2024-03-25T18:25:45.021 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.020+0000 7ff4637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:45.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.021+0000 7ff4637fe640 10 client.?.objecter ms_handle_connect 0x7ff46c0728f0 2024-03-25T18:25:45.022 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.021+0000 7ff4637fe640 10 client.?.objecter resend_mon_ops 2024-03-25T18:25:45.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.024+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2102455150 0 0) 0x7ff46c1d8340 con 0x7ff46c06f840 2024-03-25T18:25:45.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.024+0000 7ff4637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:45.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.024+0000 7ff4637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:45.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.024+0000 7ff4637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:45.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.024+0000 7ff4637fe640 10 monclient(hunting): my global_id is 4713 2024-03-25T18:25:45.026 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.024+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.7:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff44c004eb0 con 0x7ff46c06f840 2024-03-25T18:25:45.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.038+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2391495582 0 0) 0x7ff44c004eb0 con 0x7ff46c06f840 2024-03-25T18:25:45.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.038+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.7:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff46c1d8340 con 0x7ff46c06f840 2024-03-25T18:25:45.039 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.038+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff454001970 con 0x7ff46c06f840 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.038+0000 7ff4637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.038+0000 7ff4637fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.038+0000 7ff4637fe640 10 monclient(hunting): dump: 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:45.040 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.038+0000 7ff4637fe640 10 monclient(hunting): _finish_auth 0 2024-03-25T18:25:45.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.042+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.1 v1:172.21.15.183:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3341998287 0 0) 0x7ff46c1d87d0 con 0x7ff46c06f460 2024-03-25T18:25:45.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.042+0000 7ff4637fe640 10 monclient(hunting): _negotiate 2024-03-25T18:25:45.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.042+0000 7ff4637fe640 10 monclient(hunting): _init_auth method 2 2024-03-25T18:25:45.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.042+0000 7ff4637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-25T18:25:45.043 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.042+0000 7ff4637fe640 10 monclient(hunting): my global_id is 4720 2024-03-25T18:25:45.044 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.042+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.183:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff44c004eb0 con 0x7ff46c06f460 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2026865742 0 0) 0x7ff46c1d8340 con 0x7ff46c06f840 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 >> v1:172.21.15.7:6790/0 conn(0x7ff46c0728f0 legacy=0x7ff46c1e23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 >> v1:172.21.15.183:6789/0 conn(0x7ff46c06f460 legacy=0x7ff46c1d79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 10 monclient: _finish_hunting 0 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 1 monclient: found mon.a 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:45.051 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.7:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff46c136250 con 0x7ff46c06f840 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 10 monclient: _finish_auth 0 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 10 monclient: _check_auth_tickets 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4712f6640 5 monclient: authenticate success, global_id 4713 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4712f6640 10 monclient: _renew_subs 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4712f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.049+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.7:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff46c0ab140 con 0x7ff46c06f840 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 10 client.4713.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 10 monclient: _renew_subs 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.7:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff46c0ab490 con 0x7ff46c06f840 2024-03-25T18:25:45.052 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 1 librados: init done 2024-03-25T18:25:45.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:45.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:45.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:45.053 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.050+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff46c1d80b0 con 0x7ff46c06f840 2024-03-25T18:25:45.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.277+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff454003610 con 0x7ff46c06f840 2024-03-25T18:25:45.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.277+0000 7ff4637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-25T18:25:45.278 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.277+0000 7ff469ffb640 4 set_mon_vals no callback set 2024-03-25T18:25:45.371 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.370+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 1164386082 0 0) 0x7ff454003920 con 0x7ff46c06f840 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.370+0000 7ff4637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.370+0000 7ff4637fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.370+0000 7ff4637fe640 10 monclient: dump: 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:epoch 1 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:fsid 81cdc36e-13e4-4148-a153-7df7e1b15d1a 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:last_changed 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:created 2024-03-25T18:13:22.423015+0000 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:min_mon_release 19 (squid) 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:election_strategy: 3 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:0: v1:172.21.15.7:6789/0 mon.a 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:1: v1:172.21.15.183:6789/0 mon.b 2024-03-25T18:25:45.372 INFO:teuthology.misc.health.smithi007.stderr:2: v1:172.21.15.7:6790/0 mon.c 2024-03-25T18:25:45.373 INFO:teuthology.misc.health.smithi007.stderr: 2024-03-25T18:25:45.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.371+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 646560181 0 0) 0x7ff454019970 con 0x7ff46c06f840 2024-03-25T18:25:45.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.371+0000 7ff46affd640 10 monclient: get_auth_request con 0x7ff44c06ddd0 auth_method 0 2024-03-25T18:25:45.373 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.372+0000 7ff4637fe640 10 client.4713.objecter ms_handle_connect 0x7ff44c06ddd0 2024-03-25T18:25:45.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2429557061 0 0) 0x7ff4540535f0 con 0x7ff46c06f840 2024-03-25T18:25:45.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 10 client.4713.objecter ms_dispatch 0x7ff46c000b90 osd_map(11..11 src has 1..11) v4 2024-03-25T18:25:45.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 3 client.4713.objecter handle_osd_map got epochs [11,11] > 0 2024-03-25T18:25:45.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 3 client.4713.objecter handle_osd_map decoding full epoch 11 2024-03-25T18:25:45.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 20 client.4713.objecter dump_active .. 0 homeless 2024-03-25T18:25:45.380 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff45404ef30 con 0x7ff46c06f840 2024-03-25T18:25:45.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-25T18:25:45.381 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.379+0000 7ff4637fe640 10 monclient: _finish_command 1 = system:0 2024-03-25T18:25:45.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.641+0000 7ff4712f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:45.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.641+0000 7ff4712f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:45.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.641+0000 7ff4712f6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.7:6789/0 2024-03-25T18:25:45.643 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.641+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 --> v1:172.21.15.7:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff46c070960 con 0x7ff46c06f840 2024-03-25T18:25:45.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.661+0000 7ff4637fe640 1 -- 172.21.15.7:0/3119193430 <== mon.0 v1:172.21.15.7:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 226032213) 0x7ff454051030 con 0x7ff46c06f840 2024-03-25T18:25:45.662 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.661+0000 7ff4637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-25T18:25:45.663 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.661+0000 7ff4637fe640 10 monclient: _finish_command 2 = system:0 2024-03-25T18:25:45.663 INFO:teuthology.misc.health.smithi007.stdout: 2024-03-25T18:25:45.663 INFO:teuthology.misc.health.smithi007.stdout:{"epoch":11,"fsid":"81cdc36e-13e4-4148-a153-7df7e1b15d1a","created":"2024-03-25T18:14:10.074885+0000","modified":"2024-03-25T18:24:14.287145+0000","last_up_change":"0.000000","last_in_change":"2024-03-25T18:24:14.287145+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"03af994c-009d-49db-af60-10fa9c6fc71e","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"f245fce3-4660-44a7-8c53-39a3a2091c33","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"1b46f77c-50d6-4e5b-8c25-a5ae755599fc","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a0e072c8-3cd3-49ab-843e-32eef8e51002","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"d580d02b-51de-41f1-90c1-60cfcb18553e","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ee928186-6517-45e3-8997-8569eca4afbe","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"d514f04c-85ce-455d-8b9c-101dd208eb60","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"543897a0-3263-4a41-ba5a-8c3dc035283a","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-25T18:25:45.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.663+0000 7ff4712f6640 10 librados: watch_flush enter 2024-03-25T18:25:45.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.663+0000 7ff4712f6640 10 librados: watch_flush exit 2024-03-25T18:25:45.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.663+0000 7ff4712f6640 20 client.4713.objecter shutdown clearing up homeless session... 2024-03-25T18:25:45.665 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.663+0000 7ff4712f6640 10 client.4713.objecter successfully canceled tick 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.663+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 >> v1:172.21.15.7:6800/1717564767 conn(0x7ff44c06ddd0 legacy=0x7ff44c070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.663+0000 7ff4712f6640 10 monclient: shutdown 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.663+0000 7ff4712f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.664+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 >> v1:172.21.15.7:6789/0 conn(0x7ff46c06f840 legacy=0x7ff46c0aaa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.664+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 shutdown_connections 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.664+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 >> 172.21.15.7:0/3119193430 conn(0x7ff46c0b4bd0 msgr2=0x7ff46c0af4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.664+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 shutdown_connections 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.664+0000 7ff4712f6640 1 -- 172.21.15.7:0/3119193430 wait complete. 2024-03-25T18:25:45.666 INFO:teuthology.misc.health.smithi007.stderr:2024-03-25T18:25:45.664+0000 7ff4712f6640 1 librados: shutdown 2024-03-25T18:25:45.675 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-25T18:25:45.675 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-25T18:25:45.677 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-25T18:25:45.677 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-25T18:25:45.677 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:47.543 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:11:30.882 35536== 2024-03-25T18:25:47.543 INFO:tasks.ceph.osd.0.smithi007.stderr:==00:00:11:30.882 35536== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:47.579 INFO:tasks.ceph.osd.0.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:51.779 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:51.779 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.0.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 0" 2024-03-25T18:25:51.780 INFO:tasks.ceph.osd.0:Stopped 2024-03-25T18:25:51.780 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-25T18:25:51.780 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:53.824 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:11:37.159 35542== 2024-03-25T18:25:53.825 INFO:tasks.ceph.osd.1.smithi007.stderr:==00:00:11:37.159 35542== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:53.860 INFO:tasks.ceph.osd.1.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:57.882 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:25:57.882 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.1.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 1" 2024-03-25T18:25:57.883 INFO:tasks.ceph.osd.1:Stopped 2024-03-25T18:25:57.883 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-25T18:25:57.883 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:25:59.749 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:11:43.088 35531== 2024-03-25T18:25:59.750 INFO:tasks.ceph.osd.2.smithi007.stderr:==00:00:11:43.088 35531== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:25:59.782 INFO:tasks.ceph.osd.2.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:25:59.998 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:06.635 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:25:59.999 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:06.635 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.007 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:06.644 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.039 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:06.675 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.056 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:06.682 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.056 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:06.693 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.066 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:06.691 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.080 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:06.717 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:00.087 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:06.713 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:03.985 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:03.985 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.2.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 2" 2024-03-25T18:26:03.986 INFO:tasks.ceph.osd.2:Stopped 2024-03-25T18:26:03.986 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-25T18:26:03.986 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:05.842 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:11:49.175 35543== 2024-03-25T18:26:05.843 INFO:tasks.ceph.osd.3.smithi007.stderr:==00:00:11:49.175 35543== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:05.885 INFO:tasks.ceph.osd.3.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:10.088 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:10.088 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.3.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 3" 2024-03-25T18:26:10.088 INFO:tasks.ceph.osd.3:Stopped 2024-03-25T18:26:10.089 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-25T18:26:10.089 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:12.010 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:11:55.360 35650== 2024-03-25T18:26:12.011 INFO:tasks.ceph.osd.4.smithi183.stderr:==00:00:11:55.360 35650== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:12.050 INFO:tasks.ceph.osd.4.smithi183.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:16.190 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:16.191 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.4.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 4" 2024-03-25T18:26:16.191 INFO:tasks.ceph.osd.4:Stopped 2024-03-25T18:26:16.192 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-25T18:26:16.192 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:18.016 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:12:01.364 35651== 2024-03-25T18:26:18.016 INFO:tasks.ceph.osd.5.smithi183.stderr:==00:00:12:01.364 35651== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:18.054 INFO:tasks.ceph.osd.5.smithi183.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:20.036 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:26.672 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.040 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:26.677 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.067 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:26.703 35151== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.090 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:26.716 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.093 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:26.719 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.095 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:26.732 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.098 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:26.735 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.111 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:26.737 35107== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:20.116 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:26.753 35153== Warning: unimplemented fcntl command: 1036 2024-03-25T18:26:22.294 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:22.294 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.5.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 5" 2024-03-25T18:26:22.295 INFO:tasks.ceph.osd.5:Stopped 2024-03-25T18:26:22.295 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-25T18:26:22.295 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:24.118 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:12:07.465 35652== 2024-03-25T18:26:24.119 INFO:tasks.ceph.osd.6.smithi183.stderr:==00:00:12:07.465 35652== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:24.169 INFO:tasks.ceph.osd.6.smithi183.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:28.397 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:28.397 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.6.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 6" 2024-03-25T18:26:28.397 INFO:tasks.ceph.osd.6:Stopped 2024-03-25T18:26:28.398 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-25T18:26:28.398 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:29.949 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:12:13.283 35653== 2024-03-25T18:26:29.949 INFO:tasks.ceph.osd.7.smithi183.stderr:==00:00:12:13.283 35653== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:29.980 INFO:tasks.ceph.osd.7.smithi183.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:34.500 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:34.500 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/osd.7.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck ceph-osd -f --cluster ceph -i 7" 2024-03-25T18:26:34.501 INFO:tasks.ceph.osd.7:Stopped 2024-03-25T18:26:34.501 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-25T18:26:34.502 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-25T18:26:34.502 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:34.502 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:26:34.501+0000 7f3c13682640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term ceph-mgr -f --cluster ceph -i y (PID: 35158) UID: 0 2024-03-25T18:26:34.503 INFO:tasks.ceph.mgr.y.smithi007.stderr:2024-03-25T18:26:34.501+0000 7f3c13682640 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-03-25T18:26:34.529 INFO:tasks.ceph.mgr.y:Stopped 2024-03-25T18:26:34.529 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-25T18:26:34.529 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:34.557 INFO:tasks.ceph.mgr.x:Stopped 2024-03-25T18:26:34.557 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-25T18:26:34.557 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-25T18:26:34.557 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:34.590 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-25T18:26:34.574+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a (PID: 35118) UID: 0 2024-03-25T18:26:34.591 INFO:tasks.ceph.mon.a.smithi007.stderr:2024-03-25T18:26:34.576+0000 9f86640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-25T18:26:35.171 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:41.807 35151== 2024-03-25T18:26:35.171 INFO:tasks.ceph.mon.a.smithi007.stderr:==00:00:12:41.807 35151== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:35.181 INFO:tasks.ceph.mon.a.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:40.659 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:40.659 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.a.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i a" 2024-03-25T18:26:40.660 INFO:tasks.ceph.mon.a:Stopped 2024-03-25T18:26:40.661 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-25T18:26:40.661 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:40.683 INFO:tasks.ceph.mon.c.smithi007.stderr:2024-03-25T18:26:40.664+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c (PID: 35120) UID: 0 2024-03-25T18:26:40.683 INFO:tasks.ceph.mon.c.smithi007.stderr:2024-03-25T18:26:40.666+0000 9f86640 -1 mon.c@2(peon) e1 *** Got Signal Terminated *** 2024-03-25T18:26:41.221 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:47.857 35153== 2024-03-25T18:26:41.221 INFO:tasks.ceph.mon.c.smithi007.stderr:==00:00:12:47.857 35153== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:41.231 INFO:tasks.ceph.mon.c.smithi007.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:46.763 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:46.763 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi007 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.c.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i c" 2024-03-25T18:26:46.764 INFO:tasks.ceph.mon.c:Stopped 2024-03-25T18:26:46.764 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-25T18:26:46.764 INFO:teuthology.orchestra.run:waiting for 300 2024-03-25T18:26:46.780 INFO:tasks.ceph.mon.b.smithi183.stderr:2024-03-25T18:26:46.768+0000 9f86640 -1 received signal: Terminated from /usr/bin/python3 /bin/daemon-helper term env OPENSSL_ia32cap=~0x1000000000000000 valgrind --trace-children=no --child-silent-after-fork=yes --soname-synonyms=somalloc=*tcmalloc* --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b (PID: 35103) UID: 0 2024-03-25T18:26:46.781 INFO:tasks.ceph.mon.b.smithi183.stderr:2024-03-25T18:26:46.769+0000 9f86640 -1 mon.b@1(probing) e1 *** Got Signal Terminated *** 2024-03-25T18:26:47.367 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:53.993 35107== 2024-03-25T18:26:47.368 INFO:tasks.ceph.mon.b.smithi183.stderr:==00:00:12:53.993 35107== Exit program on first error (--exit-on-first-error=yes) 2024-03-25T18:26:47.378 INFO:tasks.ceph.mon.b.smithi183.stderr:daemon-helper: command failed with exit status 42 2024-03-25T18:26:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 42 2024-03-25T18:26:52.867 ERROR:teuthology.orchestra.daemon.state:Error while waiting for process to exit Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1447, in run_daemon yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/daemon/state.py", line 139, in stop run.wait([self.proc], timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 479, in wait proc.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi183 with status 42: "cd /home/ubuntu/cephtest && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term env 'OPENSSL_ia32cap=~0x1000000000000000' valgrind --trace-children=no --child-silent-after-fork=yes '--soname-synonyms=somalloc=*tcmalloc*' --num-callers=50 --suppressions=/home/ubuntu/cephtest/valgrind.supp --xml=yes --xml-file=/var/log/ceph/valgrind/mon.b.log --time-stamp=yes --vgdb=yes --exit-on-first-error=yes --error-exitcode=42 --tool=memcheck --leak-check=full --show-reachable=yes ceph-mon -f --cluster ceph -i b" 2024-03-25T18:26:52.867 INFO:tasks.ceph.mon.b:Stopped 2024-03-25T18:26:52.867 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-25T18:26:52.868 DEBUG:teuthology.orchestra.run.smithi007:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'but it is still running' | egrep -v 'objects unfound and apparently lost' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | egrep -v 'overall HEALTH_' | egrep -v '\(PG_' | egrep -v '\(MON_DOWN\)' | egrep -v '\(SLOW_OPS\)' | egrep -v 'slow request' | head -n 1 2024-03-25T18:26:52.911 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:26:52.911 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-25T18:26:53.064 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:26:53.064 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-25T18:26:53.263 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:26:53.264 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-25T18:26:53.329 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:26:53.330 DEBUG:teuthology.orchestra.run.smithi007:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-25T18:26:53.389 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:26:53.389 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-25T18:26:53.592 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:26:53.592 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-25T18:26:53.654 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:26:53.655 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-25T18:26:53.710 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:26:53.711 DEBUG:teuthology.orchestra.run.smithi183:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-25T18:26:53.778 INFO:tasks.ceph:Archiving mon data... 2024-03-25T18:26:53.780 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/data/mon.a.tgz 2024-03-25T18:26:53.780 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-25T18:26:53.798 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.ZYAlDYDlXF 2024-03-25T18:26:53.798 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.ZYAlDYDlXF 2024-03-25T18:26:53.902 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.ZYAlDYDlXF 2024-03-25T18:26:53.990 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.ZYAlDYDlXF is 136KB 2024-03-25T18:26:54.043 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.ZYAlDYDlXF 2024-03-25T18:26:54.058 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/data/mon.c.tgz 2024-03-25T18:26:54.058 DEBUG:teuthology.orchestra.run.smithi007:> mktemp 2024-03-25T18:26:54.112 INFO:teuthology.orchestra.run.smithi007.stdout:/tmp/tmp.2QcwL0HDp2 2024-03-25T18:26:54.113 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.2QcwL0HDp2 2024-03-25T18:26:54.204 DEBUG:teuthology.orchestra.run.smithi007:> sudo chmod 0666 /tmp/tmp.2QcwL0HDp2 2024-03-25T18:26:54.289 DEBUG:teuthology.orchestra.remote:smithi007:/tmp/tmp.2QcwL0HDp2 is 136KB 2024-03-25T18:26:54.339 DEBUG:teuthology.orchestra.run.smithi007:> rm -fr /tmp/tmp.2QcwL0HDp2 2024-03-25T18:26:54.353 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/data/mon.b.tgz 2024-03-25T18:26:54.353 DEBUG:teuthology.orchestra.run.smithi183:> mktemp 2024-03-25T18:26:54.371 INFO:teuthology.orchestra.run.smithi183.stdout:/tmp/tmp.vrCxNrfDxs 2024-03-25T18:26:54.371 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vrCxNrfDxs 2024-03-25T18:26:54.491 DEBUG:teuthology.orchestra.run.smithi183:> sudo chmod 0666 /tmp/tmp.vrCxNrfDxs 2024-03-25T18:26:54.579 DEBUG:teuthology.orchestra.remote:smithi183:/tmp/tmp.vrCxNrfDxs is 136KB 2024-03-25T18:26:54.630 DEBUG:teuthology.orchestra.run.smithi183:> rm -fr /tmp/tmp.vrCxNrfDxs 2024-03-25T18:26:54.645 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-25T18:26:54.645 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:26:54.647 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-25T18:26:55.034 DEBUG:tasks.ceph:valgrind exception message: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:26:55.312 INFO:tasks.ceph:Archiving crash dumps... 2024-03-25T18:26:55.314 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/remote/smithi007/crash 2024-03-25T18:26:55.314 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:26:55.342 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/remote/smithi183/crash 2024-03-25T18:26:55.343 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-25T18:26:55.370 INFO:tasks.ceph:Compressing logs... 2024-03-25T18:26:55.370 DEBUG:teuthology.orchestra.run.smithi007:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:26:55.386 DEBUG:teuthology.orchestra.run.smithi183:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39077.log 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39782.log 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40956.log 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39782.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41034.log 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39077.log.gz 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40956.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41892.log 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39782.log.gz 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41034.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40956.log.gz 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41892.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42165.log 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41034.log.gz 2024-03-25T18:26:55.460 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40917.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.41892.log.gz 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42165.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39821.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40917.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40055.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42165.log.gz 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39821.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40367.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40917.log.gz 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42243.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.39821.log.gz 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40367.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40055.log.gz 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.c.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42243.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.2.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40367.log.gz 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/mon.c.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.0.log 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42243.log.gz 2024-03-25T18:26:55.461 INFO:teuthology.orchestra.run.smithi007.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.c.log.gz 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.1.log 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.2.log: /var/log/ceph/valgrind/osd.0.log: 72.1% -- replaced with /var/log/ceph/valgrind/osd.2.log.gz 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.a.log 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.0.log.gz 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.1.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.3.log 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.1.log.gz 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37558.log 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr: 69.5% -- replaced with /var/log/ceph/valgrind/mon.a.log.gz 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/valgrind/osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40133.log 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr: 72.1% -- replaced with /var/log/ceph/valgrind/osd.3.log.gz 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.37558.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39664.log 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40133.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41073.log 2024-03-25T18:26:55.462 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.37558.log.gz 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39664.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39469.log 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40133.log.gz 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41853.log 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39664.log.gz 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39469.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41073.log.gz 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39977.log 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41853.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39469.log.gz 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40839.log 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39977.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39391.log 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.41853.log.gz 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40839.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40523.log 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39977.log.gz 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39391.log: gzip -5 --verbose 85.8% -- -- replaced with /var/log/ceph/ceph-client.admin.40839.log.gz /var/log/ceph/ceph-client.admin.39704.log 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:26:55.463 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40523.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39391.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39938.log 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40016.log 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40523.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39938.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40406.log 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39704.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40016.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39938.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40722.log 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40406.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29295.log 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40016.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40722.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40406.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40641.log 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.tmp-client.admin.29295.log: 86.8%gzip -- replaced with /var/log/ceph/ceph-client.admin.40722.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40211.log 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29295.log.gz 2024-03-25T18:26:55.464 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40641.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35570.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40641.log.gz 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40602.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40211.log.gz 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35570.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41112.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40602.log: 93.4% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.35570.log.gz 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41580.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40602.log.gz 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40800.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41112.log: /var/log/ceph/ceph-client.admin.41580.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42087.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40800.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42204.log 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41112.log.gz 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.41580.log.gz 2024-03-25T18:26:55.465 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35708.log 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.40800.log.gz 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42204.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41814.log 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.42087.log.gz 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35708.log: gzip 86.6% -5 -- replaced with /var/log/ceph/ceph-client.admin.42204.log.gz --verbose 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr: -- /var/log/ceph/ceph-client.admin.42321.log 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41814.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40289.log 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35708.log.gz 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42321.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35524.log 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41814.log.gz 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40289.log: gzip -5 --verbose 86.7% -- -- replaced with /var/log/ceph/ceph-client.admin.42321.log.gz /var/log/ceph/ceph-client.admin.42009.log 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.35524.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40289.log.gz 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-25T18:26:55.466 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42009.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41931.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.35524.log.gz 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42126.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.42009.log.gz 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41931.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40562.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42126.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.41931.log.gz 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39235.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41697.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42126.log.gz 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40562.log: /var/log/ceph/ceph-client.admin.39235.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39274.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41697.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40562.log.gz 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39352.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39235.log.gz 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39430.log 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41697.log.gz 2024-03-25T18:26:55.467 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39508.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.39274.log.gz 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40878.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39352.log.gz 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39508.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41385.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40878.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39430.log.gz 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39508.log.gz 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41385.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39116.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40878.log.gz 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41151.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41385.log.gz 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39116.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40328.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41151.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40172.log 2024-03-25T18:26:55.468 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.39116.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40328.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41970.log 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41151.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40172.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41736.log 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40328.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41970.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.40172.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41736.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39743.log 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.41970.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.41736.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39625.log 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41190.log 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39625.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40250.log 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39743.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41190.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39625.log.gz 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42048.log 2024-03-25T18:26:55.469 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41658.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40250.log: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.41190.log.gz 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42048.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39547.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40250.log.gz 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:gzip/var/log/ceph/ceph-client.admin.41658.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41463.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42048.log.gz 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39547.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38844.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41658.log.gz 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41463.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39547.log.gz 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.38844.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.36588.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41463.log.gz 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40484.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.38844.log.gz 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.36588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39899.log 2024-03-25T18:26:55.470 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40484.log: gzip 87.4% -- replaced with /var/log/ceph/ceph-client.admin.36588.log.gz 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.41619.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39899.log: gzip -5 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40484.log.gz 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr: --verbose -- /var/log/ceph/ceph-client.admin.41541.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41619.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39899.log.gz 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41541.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41619.log.gz 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41424.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40683.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.41541.log.gz 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41424.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42282.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40683.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41424.log.gz 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.42282.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40683.log.gz 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41307.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42360.log 2024-03-25T18:26:55.471 INFO:teuthology.orchestra.run.smithi007.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.42282.log.gz 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41268.log 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr: 86.3%/var/log/ceph/ceph-client.admin.42360.log: -- replaced with /var/log/ceph/ceph-client.admin.41307.log.gz 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40094.log 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.42360.log.gz 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr: 92.6% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41268.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.41268.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40995.log 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40094.log: /var/log/ceph/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39313.log 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40995.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40094.log.gz 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39860.log 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40995.log.gz 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39313.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39157.log 2024-03-25T18:26:55.472 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39860.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39313.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.39860.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41502.log 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39586.log 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41502.log: /var/log/ceph/ceph-client.admin.39157.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.39157.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41502.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40761.log 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41346.log 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.39586.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40761.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.40761.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41775.log 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41229.log 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41346.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39196.log 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.41229.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.41775.log.gz 2024-03-25T18:26:55.473 INFO:teuthology.orchestra.run.smithi007.stderr:gzip 86.7% -5 --verbose -- /var/log/ceph/ceph-client.admin.40445.log 2024-03-25T18:26:55.474 INFO:teuthology.orchestra.run.smithi007.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41229.log.gz 2024-03-25T18:26:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.39196.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-25T18:26:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-client.admin.40445.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39196.log.gz 2024-03-25T18:26:55.474 INFO:teuthology.orchestra.run.smithi007.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.40445.log.gz 2024-03-25T18:26:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35160.log 2024-03-25T18:26:55.474 INFO:teuthology.orchestra.run.smithi007.stderr:/var/log/ceph/ceph-mon.c.log: /var/log/ceph/ceph-client.admin.35160.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.35160.log.gz 2024-03-25T18:26:55.475 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-25T18:26:55.475 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35380.log 2024-03-25T18:26:55.475 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.7.log 2024-03-25T18:26:55.475 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35380.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.5.log 2024-03-25T18:26:55.475 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/valgrind/osd.7.log: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.35380.log.gz 2024-03-25T18:26:55.475 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/osd.6.log 2024-03-25T18:26:55.475 INFO:teuthology.orchestra.run.smithi183.stderr: 68.4% -- replaced with /var/log/ceph/valgrind/osd.7.log.gz 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/valgrind/mon.b.log 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/valgrind/osd.5.log: /var/log/ceph/valgrind/osd.6.log: gzip -5 --verbose -- /var/log/ceph/valgrind/osd.4.log 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.5.log.gz 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr: 72.0% -- replaced with /var/log/ceph/valgrind/osd.6.log.gz 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/valgrind/mon.b.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr: 69.6% -- replaced with /var/log/ceph/valgrind/mon.b.log.gz 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/valgrind/osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35300.log 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.audit.log: 72.0% -- replaced with /var/log/ceph/valgrind/osd.4.log.gz 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35340.log 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35300.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr: 88.1% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35340.log: gzip -5 --verbose 87.6% -- /var/log/ceph/ceph-client.admin.35500.log -- replaced with /var/log/ceph/ceph-client.admin.35300.log.gz 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-25T18:26:55.476 INFO:teuthology.orchestra.run.smithi183.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35340.log.gz 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.4.log: /var/log/ceph/ceph-client.admin.35500.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35551.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.35500.log.gz 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35551.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35260.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- 86.9% -- replaced with /var/log/ceph/ceph-client.admin.35551.log.gz 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr: /var/log/ceph/ceph.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35260.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35460.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph.log: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.35260.log.gz 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35420.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-client.admin.35420.log: /var/log/ceph/ceph-client.admin.35460.log: 87.1% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr: 87.6% 87.6% -- replaced with /var/log/ceph/ceph-client.admin.35420.log.gz 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr: -- replaced with /var/log/ceph/ceph-client.admin.35460.log.gz 2024-03-25T18:26:55.477 INFO:teuthology.orchestra.run.smithi183.stderr:/var/log/ceph/ceph-mon.b.log: 93.7% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-25T18:26:55.512 INFO:teuthology.orchestra.run.smithi007.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-25T18:26:56.263 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-25T18:26:56.297 INFO:teuthology.orchestra.run.smithi183.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-25T18:26:56.355 INFO:teuthology.orchestra.run.smithi183.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-25T18:26:56.545 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-25T18:26:56.592 INFO:teuthology.orchestra.run.smithi183.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-25T18:26:56.621 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-25T18:26:56.631 INFO:teuthology.orchestra.run.smithi183.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-25T18:26:56.641 INFO:teuthology.orchestra.run.smithi007.stderr: 92.9% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-25T18:27:00.286 INFO:teuthology.orchestra.run.smithi007.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-25T18:27:00.380 INFO:teuthology.orchestra.run.smithi183.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-25T18:27:00.382 INFO:teuthology.orchestra.run.smithi183.stderr: 2024-03-25T18:27:00.382 INFO:teuthology.orchestra.run.smithi183.stderr:real 0m4.951s 2024-03-25T18:27:00.382 INFO:teuthology.orchestra.run.smithi183.stderr:user 0m8.747s 2024-03-25T18:27:00.382 INFO:teuthology.orchestra.run.smithi183.stderr:sys 0m0.316s 2024-03-25T18:27:00.660 INFO:teuthology.orchestra.run.smithi007.stderr: 90.6% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-25T18:27:00.662 INFO:teuthology.orchestra.run.smithi007.stderr: 2024-03-25T18:27:00.662 INFO:teuthology.orchestra.run.smithi007.stderr:real 0m5.264s 2024-03-25T18:27:00.662 INFO:teuthology.orchestra.run.smithi007.stderr:user 0m14.106s 2024-03-25T18:27:00.662 INFO:teuthology.orchestra.run.smithi007.stderr:sys 0m0.505s 2024-03-25T18:27:00.663 INFO:tasks.ceph:Archiving logs... 2024-03-25T18:27:00.663 DEBUG:teuthology.misc:Transferring archived files from smithi007:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/remote/smithi007/log 2024-03-25T18:27:00.664 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:27:03.150 DEBUG:teuthology.misc:Transferring archived files from smithi183:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/remote/smithi183/log 2024-03-25T18:27:03.151 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-25T18:27:04.692 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:27:04.903 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=25aad7c7cbb24b0db4f4e76680884da6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 329, in valgrind_post yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 346, in valgrind_post raise valgrind_exception Exception: valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun 2024-03-25T18:27:04.908 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-25T18:27:04.918 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-25T18:27:04.955 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-25T18:27:04.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:27:04.970 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-25T18:27:05.044 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-25T18:27:05.053 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-25T18:27:05.053 DEBUG:teuthology.orchestra.run.smithi007:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:27:05.056 DEBUG:teuthology.orchestra.run.smithi183:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-25T18:27:05.069 INFO:teuthology.orchestra.run.smithi007.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:27:05.072 INFO:teuthology.orchestra.run.smithi183.stderr:bash: line 1: ntpq: command not found 2024-03-25T18:27:05.073 INFO:teuthology.orchestra.run.smithi007.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:27:05.073 INFO:teuthology.orchestra.run.smithi007.stdout:=============================================================================== 2024-03-25T18:27:05.073 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 43 +744us[ +347us] +/- 101ms 2024-03-25T18:27:05.074 INFO:teuthology.orchestra.run.smithi007.stdout:^* hv02.front.sepia.ceph.com 3 6 377 39 -1131us[-1529us] +/- 51ms 2024-03-25T18:27:05.074 INFO:teuthology.orchestra.run.smithi007.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 35 +845us[ +845us] +/- 71ms 2024-03-25T18:27:05.074 INFO:teuthology.orchestra.run.smithi007.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:27:05.077 INFO:teuthology.orchestra.run.smithi183.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-25T18:27:05.077 INFO:teuthology.orchestra.run.smithi183.stdout:=============================================================================== 2024-03-25T18:27:05.077 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 36 +785us[ +785us] +/- 101ms 2024-03-25T18:27:05.077 INFO:teuthology.orchestra.run.smithi183.stdout:^* hv02.front.sepia.ceph.com 3 6 377 50 -1105us[-1120us] +/- 51ms 2024-03-25T18:27:05.077 INFO:teuthology.orchestra.run.smithi183.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 44 +901us[ +901us] +/- 71ms 2024-03-25T18:27:05.078 INFO:teuthology.orchestra.run.smithi183.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-25T18:27:05.078 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-25T18:27:05.087 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-25T18:27:05.088 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-25T18:27:05.107 DEBUG:teuthology.orchestra.run.smithi007:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:27:05.187 DEBUG:teuthology.orchestra.run.smithi183:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-25T18:27:05.259 DEBUG:teuthology.orchestra.run.smithi007:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:27:05.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:27:05.291 DEBUG:teuthology.orchestra.run.smithi183:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-25T18:27:05.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:27:05.323 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-25T18:27:05.335 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-25T18:27:05.373 INFO:teuthology.task.internal:Duration was 7736.862686 seconds 2024-03-25T18:27:05.373 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-25T18:27:05.399 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-25T18:27:05.399 DEBUG:teuthology.orchestra.run.smithi007:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:27:05.402 DEBUG:teuthology.orchestra.run.smithi183:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-25T18:27:05.445 INFO:teuthology.orchestra.run.smithi007.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:27:05.446 INFO:teuthology.orchestra.run.smithi183.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-25T18:27:05.677 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-25T18:27:05.677 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi007.front.sepia.ceph.com 2024-03-25T18:27:05.677 DEBUG:teuthology.orchestra.run.smithi007:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:27:05.697 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi183.front.sepia.ceph.com 2024-03-25T18:27:05.697 DEBUG:teuthology.orchestra.run.smithi183:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-25T18:27:05.732 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-25T18:27:05.732 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:27:05.739 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-25T18:27:05.802 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-25T18:27:05.802 DEBUG:teuthology.orchestra.run.smithi007:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:27:05.808 DEBUG:teuthology.orchestra.run.smithi183:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-25T18:27:05.947 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-25T18:27:05.956 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-25T18:27:05.956 DEBUG:teuthology.orchestra.run.smithi007:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:27:05.983 DEBUG:teuthology.orchestra.run.smithi183:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-25T18:27:06.011 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-25T18:27:06.021 DEBUG:teuthology.orchestra.run.smithi007:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:27:06.026 DEBUG:teuthology.orchestra.run.smithi183:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:27:06.048 INFO:teuthology.orchestra.run.smithi007.stdout:kernel.core_pattern = core 2024-03-25T18:27:06.076 INFO:teuthology.orchestra.run.smithi183.stdout:kernel.core_pattern = core 2024-03-25T18:27:06.109 DEBUG:teuthology.orchestra.run.smithi007:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:27:06.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:27:06.155 DEBUG:teuthology.orchestra.run.smithi183:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-25T18:27:06.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-25T18:27:06.169 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-25T18:27:06.179 INFO:teuthology.task.internal:Transferring archived files... 2024-03-25T18:27:06.180 DEBUG:teuthology.misc:Transferring archived files from smithi007:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/remote/smithi007 2024-03-25T18:27:06.180 DEBUG:teuthology.orchestra.run.smithi007:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:27:06.230 DEBUG:teuthology.misc:Transferring archived files from smithi183:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262/remote/smithi183 2024-03-25T18:27:06.230 DEBUG:teuthology.orchestra.run.smithi183:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-25T18:27:06.265 INFO:teuthology.task.internal:Removing archive directory... 2024-03-25T18:27:06.266 DEBUG:teuthology.orchestra.run.smithi007:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:27:06.268 DEBUG:teuthology.orchestra.run.smithi183:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-25T18:27:06.314 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-25T18:27:06.326 INFO:teuthology.task.internal:Not uploading archives. 2024-03-25T18:27:06.326 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-25T18:27:06.341 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-25T18:27:06.342 DEBUG:teuthology.orchestra.run.smithi007:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:27:06.344 DEBUG:teuthology.orchestra.run.smithi183:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-25T18:27:06.356 INFO:teuthology.orchestra.run.smithi007.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:27 /home/ubuntu/cephtest 2024-03-25T18:27:06.370 INFO:teuthology.orchestra.run.smithi183.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 25 18:27 /home/ubuntu/cephtest 2024-03-25T18:27:06.371 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-25T18:27:06.381 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-25T18:27:06.425 INFO:teuthology.nuke:Checking targets against current locks 2024-03-25T18:27:06.451 DEBUG:teuthology.nuke:shortname: smithi007 2024-03-25T18:27:06.451 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:27:06.477 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi007.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:59.402534', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNO605m0K+s3ytSEjhPuzWOnVvx5mM9qJa4VSjBMg+mUch9L3YOnpZK3XO1/G98YOVP8dPUsCh76nbIdEPo2i1s='} 2024-03-25T18:27:06.482 DEBUG:teuthology.nuke:shortname: smithi183 2024-03-25T18:27:06.482 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-25T18:27:06.506 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi183.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-25_13:28:33-rados:verify-main-distro-default-smithi/7621262', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-25 16:09:59.404415', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCCm4RjM8duBU3f188Ut1weiOsvtzl+k29zkv296jFRM9XUdjgElzX0FBFJ83grLAIK4EwEXwcZIeVDmVJFqeMM='} 2024-03-25T18:27:06.538 INFO:teuthology.orchestra.console.smithi007:Power off 2024-03-25T18:27:06.539 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:27:06.563 DEBUG:teuthology.orchestra.console.smithi007:power off output: Chassis Power Control: Down/Off 2024-03-25T18:27:06.564 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:06.582 INFO:teuthology.orchestra.console.smithi183:Power off 2024-03-25T18:27:06.582 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-25T18:27:06.593 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-25T18:27:06.607 DEBUG:teuthology.orchestra.console.smithi183:power off output: Chassis Power Control: Down/Off 2024-03-25T18:27:06.607 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:06.631 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-25T18:27:10.595 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:10.632 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:10.706 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is on 2024-03-25T18:27:10.746 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-25T18:27:14.710 DEBUG:teuthology.orchestra.console.smithi007:pexpect command: ipmitool -H smithi007.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:14.747 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:14.824 DEBUG:teuthology.orchestra.console.smithi007:check power output: Chassis Power is off 2024-03-25T18:27:14.858 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is on 2024-03-25T18:27:14.925 INFO:teuthology.orchestra.console.smithi007:Power off completed 2024-03-25T18:27:18.862 DEBUG:teuthology.orchestra.console.smithi183:pexpect command: ipmitool -H smithi183.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-25T18:27:18.973 DEBUG:teuthology.orchestra.console.smithi183:check power output: Chassis Power is off 2024-03-25T18:27:19.074 INFO:teuthology.orchestra.console.smithi183:Power off completed 2024-03-25T18:27:19.179 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/default/{default thrashosds-health} mon_election/connectivity msgr-failures/few msgr/async-v1only objectstore/bluestore-stupid rados tasks/rados_api_tests validater/valgrind} duration: 7736.862685680389 failure_reason: 'valgrind error: Leak_StillReachable operator new[](unsigned long) UnknownInlinedFun UnknownInlinedFun' flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=25aad7c7cbb24b0db4f4e76680884da6 status: fail success: false 2024-03-25T18:27:19.180 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-25T18:27:19.254 INFO:teuthology.run:FAIL